You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/21 14:26:29 UTC

[GitHub] [beam] chamikaramj opened a new pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

chamikaramj opened a new pull request #12888:
URL: https://github.com/apache/beam/pull/12888


   This seems to be the correct place for this encoding since we would not want to override the full expansion of ReadFromPubSub/WriteToPubSub transforms.
   
   Also, simplifies PubSubWritePayload removing currently unused fields.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](htt
 ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_
 Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_P
 ostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website | Whitespace | Typescript
   --- | --- | --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/be
 am_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on a change in pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on a change in pull request #12888:
URL: https://github.com/apache/beam/pull/12888#discussion_r493650482



##########
File path: sdks/python/apache_beam/io/iobase.py
##########
@@ -992,6 +1012,43 @@ def expand(self, pcoll):
           'A sink must inherit iobase.Sink, iobase.NativeSink, '
           'or be a PTransform. Received : %r' % self.sink)
 
+  def _pubsub_write_payload(self):
+    return beam_runner_api_pb2.PubSubWritePayload(
+        topic=self.sink.full_topic,
+        id_attribute=self.sink.id_label,
+        timestamp_attribute=self.sink.timestamp_attribute)
+
+  def to_runner_api_parameter(self, context):
+    # type: (PipelineContext) -> Tuple[str, Any]
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    if isinstance(self.sink, _PubSubSink):
+      payload = self._pubsub_write_payload()
+      return (common_urns.composites.PUBSUB_WRITE.urn, payload)
+    else:
+      return super(Write, self).to_runner_api_parameter(context)
+
+  @staticmethod
+  @ptransform.PTransform.register_urn(
+      common_urns.composites.PUBSUB_WRITE.urn,
+      beam_runner_api_pb2.PubSubWritePayload)
+  def from_runner_api_parameter(ptransform, payload, unused_context):
+    # type: (Any, Any, PipelineContext) -> Write
+    if ptransform.spec.urn != common_urns.composites.PUBSUB_WRITE.urn:
+      raise ValueError(
+          'Write transform cannot be constructed for the given proto %r',
+          ptransform)
+
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    sink = _PubSubSink(
+        topic=payload.topic,
+        id_label=payload.id_attribute,
+        with_attributes=True,

Review comment:
       Hmm, actually "with_attributes=True" is not really correct for current users since we do a runner API to and from round trip here: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L511
   
   I think the next best course of action is to revert https://github.com/apache/beam/pull/12760 from the release and get this in once Boyuan's source/sink simplifications are in.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/067cba8229694e7fb9693313f51ca686746b620a?el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.34%   82.34%   -0.01%     
   ==========================================
     Files         452      453       +1     
     Lines       54016    54060      +44     
   ==========================================
   + Hits        44481    44515      +34     
   - Misses       9535     9545      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [...dks/python/apache\_beam/runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | `92.80% <0.00%> (-1.01%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.40%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `91.66% <0.00%> (ø)` | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.32%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [067cba8...fac493f](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-697113755


   Run Python PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on a change in pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on a change in pull request #12888:
URL: https://github.com/apache/beam/pull/12888#discussion_r492976506



##########
File path: sdks/python/apache_beam/io/iobase.py
##########
@@ -992,6 +1012,43 @@ def expand(self, pcoll):
           'A sink must inherit iobase.Sink, iobase.NativeSink, '
           'or be a PTransform. Received : %r' % self.sink)
 
+  def _pubsub_write_payload(self):
+    return beam_runner_api_pb2.PubSubWritePayload(
+        topic=self.sink.full_topic,
+        id_attribute=self.sink.id_label,
+        timestamp_attribute=self.sink.timestamp_attribute)
+
+  def to_runner_api_parameter(self, context):
+    # type: (PipelineContext) -> Tuple[str, Any]
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    if isinstance(self.sink, _PubSubSink):
+      payload = self._pubsub_write_payload()
+      return (common_urns.composites.PUBSUB_WRITE.urn, payload)
+    else:
+      return super(Write, self).to_runner_api_parameter(context)
+
+  @staticmethod
+  @ptransform.PTransform.register_urn(
+      common_urns.composites.PUBSUB_WRITE.urn,
+      beam_runner_api_pb2.PubSubWritePayload)
+  def from_runner_api_parameter(ptransform, payload, unused_context):
+    # type: (Any, Any, PipelineContext) -> Write
+    if ptransform.spec.urn != common_urns.composites.PUBSUB_WRITE.urn:
+      raise ValueError(
+          'Write transform cannot be constructed for the given proto %r',
+          ptransform)
+
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    sink = _PubSubSink(
+        topic=payload.topic,
+        id_label=payload.id_attribute,
+        with_attributes=True,

Review comment:
       So, I thought that's the simplification you are working on, right ? I'm fine with these properties being removed form source/sink if they are always true.
   
   This change simply updates protos (and corresponding encoding in Read/Write transforms) to match your updates to PubSub native source/sink. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on a change in pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on a change in pull request #12888:
URL: https://github.com/apache/beam/pull/12888#discussion_r493800318



##########
File path: sdks/python/apache_beam/io/iobase.py
##########
@@ -992,6 +1012,43 @@ def expand(self, pcoll):
           'A sink must inherit iobase.Sink, iobase.NativeSink, '
           'or be a PTransform. Received : %r' % self.sink)
 
+  def _pubsub_write_payload(self):
+    return beam_runner_api_pb2.PubSubWritePayload(
+        topic=self.sink.full_topic,
+        id_attribute=self.sink.id_label,
+        timestamp_attribute=self.sink.timestamp_attribute)
+
+  def to_runner_api_parameter(self, context):
+    # type: (PipelineContext) -> Tuple[str, Any]
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    if isinstance(self.sink, _PubSubSink):
+      payload = self._pubsub_write_payload()
+      return (common_urns.composites.PUBSUB_WRITE.urn, payload)
+    else:
+      return super(Write, self).to_runner_api_parameter(context)
+
+  @staticmethod
+  @ptransform.PTransform.register_urn(
+      common_urns.composites.PUBSUB_WRITE.urn,
+      beam_runner_api_pb2.PubSubWritePayload)
+  def from_runner_api_parameter(ptransform, payload, unused_context):
+    # type: (Any, Any, PipelineContext) -> Write
+    if ptransform.spec.urn != common_urns.composites.PUBSUB_WRITE.urn:
+      raise ValueError(
+          'Write transform cannot be constructed for the given proto %r',
+          ptransform)
+
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    sink = _PubSubSink(
+        topic=payload.topic,
+        id_label=payload.id_attribute,
+        with_attributes=True,

Review comment:
       I think the crux of the matter is that runners (both Dataflow and Direct) depend on pipeline->proto->pipleline transformation to preserve state. 
   All runners depend on it here: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline.py#L514
   Additional Dataflow performs a second round trip here: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L511
   
   Additionally DirectRunner refers to 'with_attributes' property here which is expected to be preserved in such a transformation: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/direct/direct_runner.py#L406
   
   My proposal it to remove other Dataflow specific state from the proto and keep `with_attributes` which is needed to preserve state of the transforms today. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696556942






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] boyuanzz commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

boyuanzz commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696410670


   Discussed with Luke and Cham separately, we can get ride of `with_attributes` and `serialized_attribute_fn ` from both Read and Write. There are corresponding changes for Write: https://github.com/apache/beam/pull/12806. We should also similar changes to Read.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-697727612






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj merged pull request #12888: [BEAM-10861]Adds PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj merged pull request #12888:
URL: https://github.com/apache/beam/pull/12888


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696151452


   R: @boyuanzz @lukecwik 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on a change in pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on a change in pull request #12888:
URL: https://github.com/apache/beam/pull/12888#discussion_r493850984



##########
File path: sdks/python/apache_beam/io/iobase.py
##########
@@ -992,6 +1012,43 @@ def expand(self, pcoll):
           'A sink must inherit iobase.Sink, iobase.NativeSink, '
           'or be a PTransform. Received : %r' % self.sink)
 
+  def _pubsub_write_payload(self):
+    return beam_runner_api_pb2.PubSubWritePayload(
+        topic=self.sink.full_topic,
+        id_attribute=self.sink.id_label,
+        timestamp_attribute=self.sink.timestamp_attribute)
+
+  def to_runner_api_parameter(self, context):
+    # type: (PipelineContext) -> Tuple[str, Any]
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    if isinstance(self.sink, _PubSubSink):
+      payload = self._pubsub_write_payload()
+      return (common_urns.composites.PUBSUB_WRITE.urn, payload)
+    else:
+      return super(Write, self).to_runner_api_parameter(context)
+
+  @staticmethod
+  @ptransform.PTransform.register_urn(
+      common_urns.composites.PUBSUB_WRITE.urn,
+      beam_runner_api_pb2.PubSubWritePayload)
+  def from_runner_api_parameter(ptransform, payload, unused_context):
+    # type: (Any, Any, PipelineContext) -> Write
+    if ptransform.spec.urn != common_urns.composites.PUBSUB_WRITE.urn:
+      raise ValueError(
+          'Write transform cannot be constructed for the given proto %r',
+          ptransform)
+
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    sink = _PubSubSink(
+        topic=payload.topic,
+        id_label=payload.id_attribute,
+        with_attributes=True,

Review comment:
       After an offline discussion, we'll get the revert (https://github.com/apache/beam/pull/12914) in the unblock the 2.25.0 release and get this PR in after Boyuan's upcoming refactoring efforts.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on a change in pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on a change in pull request #12888:
URL: https://github.com/apache/beam/pull/12888#discussion_r492976506



##########
File path: sdks/python/apache_beam/io/iobase.py
##########
@@ -992,6 +1012,43 @@ def expand(self, pcoll):
           'A sink must inherit iobase.Sink, iobase.NativeSink, '
           'or be a PTransform. Received : %r' % self.sink)
 
+  def _pubsub_write_payload(self):
+    return beam_runner_api_pb2.PubSubWritePayload(
+        topic=self.sink.full_topic,
+        id_attribute=self.sink.id_label,
+        timestamp_attribute=self.sink.timestamp_attribute)
+
+  def to_runner_api_parameter(self, context):
+    # type: (PipelineContext) -> Tuple[str, Any]
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    if isinstance(self.sink, _PubSubSink):
+      payload = self._pubsub_write_payload()
+      return (common_urns.composites.PUBSUB_WRITE.urn, payload)
+    else:
+      return super(Write, self).to_runner_api_parameter(context)
+
+  @staticmethod
+  @ptransform.PTransform.register_urn(
+      common_urns.composites.PUBSUB_WRITE.urn,
+      beam_runner_api_pb2.PubSubWritePayload)
+  def from_runner_api_parameter(ptransform, payload, unused_context):
+    # type: (Any, Any, PipelineContext) -> Write
+    if ptransform.spec.urn != common_urns.composites.PUBSUB_WRITE.urn:
+      raise ValueError(
+          'Write transform cannot be constructed for the given proto %r',
+          ptransform)
+
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    sink = _PubSubSink(
+        topic=payload.topic,
+        id_label=payload.id_attribute,
+        with_attributes=True,

Review comment:
       So, I thought that's the simplification you are working on, right ? I'm fine with these properties being removed form source/sink if they are always true.
   
   This change simply updates protos (and corresponding encoding in Read/Write transforms) to match your updates to PubSub native source/sink. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696857697


   Retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/067cba8229694e7fb9693313f51ca686746b620a?el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.34%   82.34%   -0.01%     
   ==========================================
     Files         452      453       +1     
     Lines       54016    54060      +44     
   ==========================================
   + Hits        44481    44515      +34     
   - Misses       9535     9545      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [...dks/python/apache\_beam/runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | `92.80% <0.00%> (-1.01%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.40%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `91.66% <0.00%> (ø)` | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.32%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [067cba8...fac493f](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-697042667


   Thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) will **decrease** coverage by `0.06%`.
   > The diff coverage is `92.59%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.48%   82.42%   -0.07%     
   ==========================================
     Files         455      454       -1     
     Lines       54876    54349     -527     
   ==========================================
   - Hits        45266    44795     -471     
   + Misses       9610     9554      -56     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (ø)` | |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.98% <96.00%> (+0.95%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (-54.35%)` | :arrow_down: |
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `83.33% <0.00%> (-9.53%)` | :arrow_down: |
   | [sdks/python/apache\_beam/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vX19pbml0X18ucHk=) | `80.00% <0.00%> (-5.72%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <0.00%> (-3.51%)` | :arrow_down: |
   | [...ython/apache\_beam/io/gcp/tests/bigquery\_matcher.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Rlc3RzL2JpZ3F1ZXJ5X21hdGNoZXIucHk=) | `79.83% <0.00%> (-2.87%)` | :arrow_down: |
   | [conftest.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-Y29uZnRlc3QucHk=) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
   | [...e\_beam/io/gcp/big\_query\_query\_to\_table\_pipeline.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ19xdWVyeV9xdWVyeV90b190YWJsZV9waXBlbGluZS5weQ==) | `29.03% <0.00%> (-2.01%)` | :arrow_down: |
   | [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `88.28% <0.00%> (-1.52%)` | :arrow_down: |
   | ... and [57 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [5928e8b...f4b9d23](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2ff8337aa943623dfaabaf8b42876b4d9ee1c963?el=desc) will **increase** coverage by `0.03%`.
   > The diff coverage is `98.21%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   + Coverage   82.32%   82.35%   +0.03%     
   ==========================================
     Files         454      454              
     Lines       54088    54125      +37     
   ==========================================
   + Hits        44528    44575      +47     
   + Misses       9560     9550      -10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.84% <100.00%> (+1.77%)` | :arrow_up: |
   | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `93.38% <0.00%> (-0.74%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | ... and [5 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [89c3f51...98defc3](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Adds PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-707993140


   Thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/dad847069ed37ab060b7e505e0a4251aad547e80?el=desc) will **increase** coverage by `0.09%`.
   > The diff coverage is `88.74%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   + Coverage   82.32%   82.42%   +0.09%     
   ==========================================
     Files         454      454              
     Lines       54061    54349     +288     
   ==========================================
   + Hits        44508    44795     +287     
   - Misses       9553     9554       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `79.78% <44.44%> (+<0.01%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `89.97% <58.33%> (-0.63%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.98% <96.00%> (+0.65%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.84% <100.00%> (+1.77%)` | :arrow_up: |
   | [.../python/apache\_beam/io/gcp/bigquery\_io\_metadata.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2lvX21ldGFkYXRhLnB5) | `90.62% <100.00%> (+3.66%)` | :arrow_up: |
   | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
   | ... and [25 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [89c3f51...8a05b65](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/067cba8229694e7fb9693313f51ca686746b620a?el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.34%   82.34%   -0.01%     
   ==========================================
     Files         452      453       +1     
     Lines       54016    54060      +44     
   ==========================================
   + Hits        44481    44515      +34     
   - Misses       9535     9545      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [...dks/python/apache\_beam/runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | `92.80% <0.00%> (-1.01%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.40%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `91.66% <0.00%> (ø)` | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.32%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [067cba8...fac493f](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] boyuanzz commented on a change in pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

boyuanzz commented on a change in pull request #12888:
URL: https://github.com/apache/beam/pull/12888#discussion_r493000543



##########
File path: sdks/python/apache_beam/io/iobase.py
##########
@@ -992,6 +1012,43 @@ def expand(self, pcoll):
           'A sink must inherit iobase.Sink, iobase.NativeSink, '
           'or be a PTransform. Received : %r' % self.sink)
 
+  def _pubsub_write_payload(self):
+    return beam_runner_api_pb2.PubSubWritePayload(
+        topic=self.sink.full_topic,
+        id_attribute=self.sink.id_label,
+        timestamp_attribute=self.sink.timestamp_attribute)
+
+  def to_runner_api_parameter(self, context):
+    # type: (PipelineContext) -> Tuple[str, Any]
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    if isinstance(self.sink, _PubSubSink):
+      payload = self._pubsub_write_payload()
+      return (common_urns.composites.PUBSUB_WRITE.urn, payload)
+    else:
+      return super(Write, self).to_runner_api_parameter(context)
+
+  @staticmethod
+  @ptransform.PTransform.register_urn(
+      common_urns.composites.PUBSUB_WRITE.urn,
+      beam_runner_api_pb2.PubSubWritePayload)
+  def from_runner_api_parameter(ptransform, payload, unused_context):
+    # type: (Any, Any, PipelineContext) -> Write
+    if ptransform.spec.urn != common_urns.composites.PUBSUB_WRITE.urn:
+      raise ValueError(
+          'Write transform cannot be constructed for the given proto %r',
+          ptransform)
+
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    sink = _PubSubSink(
+        topic=payload.topic,
+        id_label=payload.id_attribute,
+        with_attributes=True,

Review comment:
       I only did partial cleanup to Write. We can have a followup PR to add more cleanup.
   cc: @lukecwik 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/dad847069ed37ab060b7e505e0a4251aad547e80?el=desc) will **increase** coverage by `0.24%`.
   > The diff coverage is `88.88%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   + Coverage   82.32%   82.57%   +0.24%     
   ==========================================
     Files         454      454              
     Lines       54061    55323    +1262     
   ==========================================
   + Hits        44508    45682    +1174     
   - Misses       9553     9641      +88     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `79.78% <44.44%> (+<0.01%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `89.97% <58.33%> (-0.63%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.84% <100.00%> (+1.77%)` | :arrow_up: |
   | [.../python/apache\_beam/io/gcp/bigquery\_io\_metadata.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2lvX21ldGFkYXRhLnB5) | `90.62% <100.00%> (+3.66%)` | :arrow_up: |
   | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
   | ... and [18 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [89c3f51...3162c68](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-697033693


   Run Python PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696556942


   Simplified read/write payloads as discussed. PTAL.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/067cba8229694e7fb9693313f51ca686746b620a?el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.34%   82.34%   -0.01%     
   ==========================================
     Files         452      453       +1     
     Lines       54016    54060      +44     
   ==========================================
   + Hits        44481    44515      +34     
   - Misses       9535     9545      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [...dks/python/apache\_beam/runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | `92.80% <0.00%> (-1.01%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.40%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `91.66% <0.00%> (ø)` | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.32%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [067cba8...fac493f](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696151452


   R: @boyuanzz @lukecwik 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/067cba8229694e7fb9693313f51ca686746b620a?el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.34%   82.34%   -0.01%     
   ==========================================
     Files         452      453       +1     
     Lines       54016    54060      +44     
   ==========================================
   + Hits        44481    44515      +34     
   - Misses       9535     9545      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [...dks/python/apache\_beam/runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | `92.80% <0.00%> (-1.01%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.40%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `91.66% <0.00%> (ø)` | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.32%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [067cba8...fac493f](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2ff8337aa943623dfaabaf8b42876b4d9ee1c963?el=desc) will **increase** coverage by `0.03%`.
   > The diff coverage is `98.21%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   + Coverage   82.32%   82.35%   +0.03%     
   ==========================================
     Files         454      454              
     Lines       54088    54125      +37     
   ==========================================
   + Hits        44528    44575      +47     
   + Misses       9560     9550      -10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.84% <100.00%> (+1.77%)` | :arrow_up: |
   | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `93.38% <0.00%> (-0.74%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | ... and [5 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [89c3f51...98defc3](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/dad847069ed37ab060b7e505e0a4251aad547e80?el=desc) will **increase** coverage by `0.24%`.
   > The diff coverage is `88.88%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   + Coverage   82.32%   82.57%   +0.24%     
   ==========================================
     Files         454      454              
     Lines       54061    55323    +1262     
   ==========================================
   + Hits        44508    45682    +1174     
   - Misses       9553     9641      +88     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `79.78% <44.44%> (+<0.01%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `89.97% <58.33%> (-0.63%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.84% <100.00%> (+1.77%)` | :arrow_up: |
   | [.../python/apache\_beam/io/gcp/bigquery\_io\_metadata.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2lvX21ldGFkYXRhLnB5) | `90.62% <100.00%> (+3.66%)` | :arrow_up: |
   | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
   | ... and [18 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [89c3f51...98defc3](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Adds PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-707858355


   @boyuanzz Updated the PR as discussed offline. PTAL.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2ff8337aa943623dfaabaf8b42876b4d9ee1c963?el=desc) will **increase** coverage by `0.03%`.
   > The diff coverage is `98.21%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   + Coverage   82.32%   82.35%   +0.03%     
   ==========================================
     Files         454      454              
     Lines       54088    54125      +37     
   ==========================================
   + Hits        44528    44575      +47     
   + Misses       9560     9550      -10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.84% <100.00%> (+1.77%)` | :arrow_up: |
   | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `93.38% <0.00%> (-0.74%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | ... and [5 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [89c3f51...98defc3](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] boyuanzz commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

boyuanzz commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696410670


   Discussed with Luke and Cham separately, we can get ride of `with_attributes` and `serialized_attribute_fn ` from both Read and Write. There are corresponding changes for Write: https://github.com/apache/beam/pull/12806. We should also similar changes to Read.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/067cba8229694e7fb9693313f51ca686746b620a?el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.34%   82.34%   -0.01%     
   ==========================================
     Files         452      453       +1     
     Lines       54016    54060      +44     
   ==========================================
   + Hits        44481    44515      +34     
   - Misses       9535     9545      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [...dks/python/apache\_beam/runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | `92.80% <0.00%> (-1.01%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.40%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `91.66% <0.00%> (ø)` | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.32%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [067cba8...fac493f](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2ff8337aa943623dfaabaf8b42876b4d9ee1c963?el=desc) will **increase** coverage by `0.03%`.
   > The diff coverage is `98.21%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   + Coverage   82.32%   82.35%   +0.03%     
   ==========================================
     Files         454      454              
     Lines       54088    54125      +37     
   ==========================================
   + Hits        44528    44575      +47     
   + Misses       9560     9550      -10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.84% <100.00%> (+1.77%)` | :arrow_up: |
   | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `93.38% <0.00%> (-0.74%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | ... and [5 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [89c3f51...98defc3](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/dad847069ed37ab060b7e505e0a4251aad547e80?el=desc) will **increase** coverage by `0.24%`.
   > The diff coverage is `88.88%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   + Coverage   82.32%   82.57%   +0.24%     
   ==========================================
     Files         454      454              
     Lines       54061    55323    +1262     
   ==========================================
   + Hits        44508    45682    +1174     
   - Misses       9553     9641      +88     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `79.78% <44.44%> (+<0.01%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `89.97% <58.33%> (-0.63%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.84% <100.00%> (+1.77%)` | :arrow_up: |
   | [.../python/apache\_beam/io/gcp/bigquery\_io\_metadata.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2lvX21ldGFkYXRhLnB5) | `90.62% <100.00%> (+3.66%)` | :arrow_up: |
   | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
   | ... and [18 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [89c3f51...8a05b65](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] boyuanzz commented on a change in pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

boyuanzz commented on a change in pull request #12888:
URL: https://github.com/apache/beam/pull/12888#discussion_r492971691



##########
File path: sdks/python/apache_beam/io/iobase.py
##########
@@ -992,6 +1012,43 @@ def expand(self, pcoll):
           'A sink must inherit iobase.Sink, iobase.NativeSink, '
           'or be a PTransform. Received : %r' % self.sink)
 
+  def _pubsub_write_payload(self):
+    return beam_runner_api_pb2.PubSubWritePayload(
+        topic=self.sink.full_topic,
+        id_attribute=self.sink.id_label,
+        timestamp_attribute=self.sink.timestamp_attribute)
+
+  def to_runner_api_parameter(self, context):
+    # type: (PipelineContext) -> Tuple[str, Any]
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    if isinstance(self.sink, _PubSubSink):
+      payload = self._pubsub_write_payload()
+      return (common_urns.composites.PUBSUB_WRITE.urn, payload)
+    else:
+      return super(Write, self).to_runner_api_parameter(context)
+
+  @staticmethod
+  @ptransform.PTransform.register_urn(
+      common_urns.composites.PUBSUB_WRITE.urn,
+      beam_runner_api_pb2.PubSubWritePayload)
+  def from_runner_api_parameter(ptransform, payload, unused_context):
+    # type: (Any, Any, PipelineContext) -> Write
+    if ptransform.spec.urn != common_urns.composites.PUBSUB_WRITE.urn:
+      raise ValueError(
+          'Write transform cannot be constructed for the given proto %r',
+          ptransform)
+
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    sink = _PubSubSink(
+        topic=payload.topic,
+        id_label=payload.id_attribute,
+        with_attributes=True,

Review comment:
       If `with_attributes` is always `True`, should we also remove `with_attributes` from `_PubSubSink` and  `_PubSubSource`? This change requires `_PubSubSource` only outputs `PubSubMessage` and `_PubSubSink` only accepts `PubSubMessage`. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Adds PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) will **decrease** coverage by `0.06%`.
   > The diff coverage is `92.59%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.48%   82.42%   -0.07%     
   ==========================================
     Files         455      454       -1     
     Lines       54876    54349     -527     
   ==========================================
   - Hits        45266    44795     -471     
   + Misses       9610     9554      -56     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (ø)` | |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.98% <96.00%> (+0.95%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (-54.35%)` | :arrow_down: |
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `83.33% <0.00%> (-9.53%)` | :arrow_down: |
   | [sdks/python/apache\_beam/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vX19pbml0X18ucHk=) | `80.00% <0.00%> (-5.72%)` | :arrow_down: |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <0.00%> (-3.51%)` | :arrow_down: |
   | [...ython/apache\_beam/io/gcp/tests/bigquery\_matcher.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Rlc3RzL2JpZ3F1ZXJ5X21hdGNoZXIucHk=) | `79.83% <0.00%> (-2.87%)` | :arrow_down: |
   | [conftest.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-Y29uZnRlc3QucHk=) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
   | [...e\_beam/io/gcp/big\_query\_query\_to\_table\_pipeline.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ19xdWVyeV9xdWVyeV90b190YWJsZV9waXBlbGluZS5weQ==) | `29.03% <0.00%> (-2.01%)` | :arrow_down: |
   | [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `88.28% <0.00%> (-1.52%)` | :arrow_down: |
   | ... and [57 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [5928e8b...980b065](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on a change in pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on a change in pull request #12888:
URL: https://github.com/apache/beam/pull/12888#discussion_r493728664



##########
File path: sdks/python/apache_beam/io/iobase.py
##########
@@ -992,6 +1012,43 @@ def expand(self, pcoll):
           'A sink must inherit iobase.Sink, iobase.NativeSink, '
           'or be a PTransform. Received : %r' % self.sink)
 
+  def _pubsub_write_payload(self):
+    return beam_runner_api_pb2.PubSubWritePayload(
+        topic=self.sink.full_topic,
+        id_attribute=self.sink.id_label,
+        timestamp_attribute=self.sink.timestamp_attribute)
+
+  def to_runner_api_parameter(self, context):
+    # type: (PipelineContext) -> Tuple[str, Any]
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    if isinstance(self.sink, _PubSubSink):
+      payload = self._pubsub_write_payload()
+      return (common_urns.composites.PUBSUB_WRITE.urn, payload)
+    else:
+      return super(Write, self).to_runner_api_parameter(context)
+
+  @staticmethod
+  @ptransform.PTransform.register_urn(
+      common_urns.composites.PUBSUB_WRITE.urn,
+      beam_runner_api_pb2.PubSubWritePayload)
+  def from_runner_api_parameter(ptransform, payload, unused_context):
+    # type: (Any, Any, PipelineContext) -> Write
+    if ptransform.spec.urn != common_urns.composites.PUBSUB_WRITE.urn:
+      raise ValueError(
+          'Write transform cannot be constructed for the given proto %r',
+          ptransform)
+
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    sink = _PubSubSink(
+        topic=payload.topic,
+        id_label=payload.id_attribute,
+        with_attributes=True,

Review comment:
       I added 'with_attributes' property back. That should fix the failure here. We can later deprecate/remove it from the proto if we truly remove all it's usage including DirectRunner.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/067cba8229694e7fb9693313f51ca686746b620a?el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.34%   82.34%   -0.01%     
   ==========================================
     Files         452      453       +1     
     Lines       54016    54060      +44     
   ==========================================
   + Hits        44481    44515      +34     
   - Misses       9535     9545      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [...dks/python/apache\_beam/runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | `92.80% <0.00%> (-1.01%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.40%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `91.66% <0.00%> (ø)` | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.32%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [067cba8...fac493f](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] boyuanzz commented on a change in pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

boyuanzz commented on a change in pull request #12888:
URL: https://github.com/apache/beam/pull/12888#discussion_r493782709



##########
File path: sdks/python/apache_beam/io/iobase.py
##########
@@ -992,6 +1012,43 @@ def expand(self, pcoll):
           'A sink must inherit iobase.Sink, iobase.NativeSink, '
           'or be a PTransform. Received : %r' % self.sink)
 
+  def _pubsub_write_payload(self):
+    return beam_runner_api_pb2.PubSubWritePayload(
+        topic=self.sink.full_topic,
+        id_attribute=self.sink.id_label,
+        timestamp_attribute=self.sink.timestamp_attribute)
+
+  def to_runner_api_parameter(self, context):
+    # type: (PipelineContext) -> Tuple[str, Any]
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    if isinstance(self.sink, _PubSubSink):
+      payload = self._pubsub_write_payload()
+      return (common_urns.composites.PUBSUB_WRITE.urn, payload)
+    else:
+      return super(Write, self).to_runner_api_parameter(context)
+
+  @staticmethod
+  @ptransform.PTransform.register_urn(
+      common_urns.composites.PUBSUB_WRITE.urn,
+      beam_runner_api_pb2.PubSubWritePayload)
+  def from_runner_api_parameter(ptransform, payload, unused_context):
+    # type: (Any, Any, PipelineContext) -> Write
+    if ptransform.spec.urn != common_urns.composites.PUBSUB_WRITE.urn:
+      raise ValueError(
+          'Write transform cannot be constructed for the given proto %r',
+          ptransform)
+
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    sink = _PubSubSink(
+        topic=payload.topic,
+        id_label=payload.id_attribute,
+        with_attributes=True,

Review comment:
       Could you explain more about why it also affects DirecrRunner? I would prefer checking in cleanup for Sink/Source first then checking in clean version of proto. What do you think?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/dad847069ed37ab060b7e505e0a4251aad547e80?el=desc) will **increase** coverage by `0.24%`.
   > The diff coverage is `88.88%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   + Coverage   82.32%   82.57%   +0.24%     
   ==========================================
     Files         454      454              
     Lines       54061    55323    +1262     
   ==========================================
   + Hits        44508    45682    +1174     
   - Misses       9553     9641      +88     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `79.78% <44.44%> (+<0.01%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `89.97% <58.33%> (-0.63%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.84% <100.00%> (+1.77%)` | :arrow_up: |
   | [.../python/apache\_beam/io/gcp/bigquery\_io\_metadata.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2lvX21ldGFkYXRhLnB5) | `90.62% <100.00%> (+3.66%)` | :arrow_up: |
   | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
   | ... and [18 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [89c3f51...98defc3](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-697076323


   Run Python PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] boyuanzz commented on a change in pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

boyuanzz commented on a change in pull request #12888:
URL: https://github.com/apache/beam/pull/12888#discussion_r492971691



##########
File path: sdks/python/apache_beam/io/iobase.py
##########
@@ -992,6 +1012,43 @@ def expand(self, pcoll):
           'A sink must inherit iobase.Sink, iobase.NativeSink, '
           'or be a PTransform. Received : %r' % self.sink)
 
+  def _pubsub_write_payload(self):
+    return beam_runner_api_pb2.PubSubWritePayload(
+        topic=self.sink.full_topic,
+        id_attribute=self.sink.id_label,
+        timestamp_attribute=self.sink.timestamp_attribute)
+
+  def to_runner_api_parameter(self, context):
+    # type: (PipelineContext) -> Tuple[str, Any]
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    if isinstance(self.sink, _PubSubSink):
+      payload = self._pubsub_write_payload()
+      return (common_urns.composites.PUBSUB_WRITE.urn, payload)
+    else:
+      return super(Write, self).to_runner_api_parameter(context)
+
+  @staticmethod
+  @ptransform.PTransform.register_urn(
+      common_urns.composites.PUBSUB_WRITE.urn,
+      beam_runner_api_pb2.PubSubWritePayload)
+  def from_runner_api_parameter(ptransform, payload, unused_context):
+    # type: (Any, Any, PipelineContext) -> Write
+    if ptransform.spec.urn != common_urns.composites.PUBSUB_WRITE.urn:
+      raise ValueError(
+          'Write transform cannot be constructed for the given proto %r',
+          ptransform)
+
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    sink = _PubSubSink(
+        topic=payload.topic,
+        id_label=payload.id_attribute,
+        with_attributes=True,

Review comment:
       If `with_attributes` is always `True`, should we also remove `with_attributes` from `_PubSubSink` and  `_PubSubSource`? This change requires `_PubSubSource` only outputs `PubSubMessage` and `_PubSubSink` only accepts `PubSubMessage`. 

##########
File path: sdks/python/apache_beam/io/iobase.py
##########
@@ -992,6 +1012,43 @@ def expand(self, pcoll):
           'A sink must inherit iobase.Sink, iobase.NativeSink, '
           'or be a PTransform. Received : %r' % self.sink)
 
+  def _pubsub_write_payload(self):
+    return beam_runner_api_pb2.PubSubWritePayload(
+        topic=self.sink.full_topic,
+        id_attribute=self.sink.id_label,
+        timestamp_attribute=self.sink.timestamp_attribute)
+
+  def to_runner_api_parameter(self, context):
+    # type: (PipelineContext) -> Tuple[str, Any]
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    if isinstance(self.sink, _PubSubSink):
+      payload = self._pubsub_write_payload()
+      return (common_urns.composites.PUBSUB_WRITE.urn, payload)
+    else:
+      return super(Write, self).to_runner_api_parameter(context)
+
+  @staticmethod
+  @ptransform.PTransform.register_urn(
+      common_urns.composites.PUBSUB_WRITE.urn,
+      beam_runner_api_pb2.PubSubWritePayload)
+  def from_runner_api_parameter(ptransform, payload, unused_context):
+    # type: (Any, Any, PipelineContext) -> Write
+    if ptransform.spec.urn != common_urns.composites.PUBSUB_WRITE.urn:
+      raise ValueError(
+          'Write transform cannot be constructed for the given proto %r',
+          ptransform)
+
+    # Importing locally to prevent circular dependencies.
+    from apache_beam.io.gcp.pubsub import _PubSubSink
+    sink = _PubSubSink(
+        topic=payload.topic,
+        id_label=payload.id_attribute,
+        with_attributes=True,

Review comment:
       I only did partial cleanup to Write. We can have a followup PR to add more cleanup.
   cc: @lukecwik 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2ff8337aa943623dfaabaf8b42876b4d9ee1c963?el=desc) will **increase** coverage by `0.03%`.
   > The diff coverage is `98.21%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   + Coverage   82.32%   82.35%   +0.03%     
   ==========================================
     Files         454      454              
     Lines       54088    54125      +37     
   ==========================================
   + Hits        44528    44575      +47     
   + Misses       9560     9550      -10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.84% <100.00%> (+1.77%)` | :arrow_up: |
   | [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `93.38% <0.00%> (-0.74%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | ... and [5 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [89c3f51...98defc3](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Adds PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-707880132


   Run Python_PVR_Flink PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] boyuanzz commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

boyuanzz commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696410670


   Discussed with Luke and Cham separately, we can get ride of `with_attributes` and `serialized_attribute_fn ` from both Read and Write. There are corresponding changes for Write: https://github.com/apache/beam/pull/12806. We should also similar changes to Read.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-697016318


   Run Python PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-697113814


   Failures are unrelated so trying again.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] boyuanzz commented on pull request #12888: [BEAM-10861]Adds PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

boyuanzz commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-707985010


   Discussed with Cham offline. In order to unblock Cham, we decided to check in this PR first and do the cleanup later when UW release is ready.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/067cba8229694e7fb9693313f51ca686746b620a?el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.34%   82.34%   -0.01%     
   ==========================================
     Files         452      453       +1     
     Lines       54016    54060      +44     
   ==========================================
   + Hits        44481    44515      +34     
   - Misses       9535     9545      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [...dks/python/apache\_beam/runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | `92.80% <0.00%> (-1.01%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.40%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `91.66% <0.00%> (ø)` | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.32%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [067cba8...fac493f](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] commented on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/067cba8229694e7fb9693313f51ca686746b620a?el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.34%   82.34%   -0.01%     
   ==========================================
     Files         452      453       +1     
     Lines       54016    54060      +44     
   ==========================================
   + Hits        44481    44515      +34     
   - Misses       9535     9545      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [...dks/python/apache\_beam/runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | `92.80% <0.00%> (-1.01%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.40%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `91.66% <0.00%> (ø)` | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.32%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [067cba8...fac493f](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/067cba8229694e7fb9693313f51ca686746b620a?el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.34%   82.34%   -0.01%     
   ==========================================
     Files         452      453       +1     
     Lines       54016    54060      +44     
   ==========================================
   + Hits        44481    44515      +34     
   - Misses       9535     9545      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [...dks/python/apache\_beam/runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | `92.80% <0.00%> (-1.01%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.40%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `91.66% <0.00%> (ø)` | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.32%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [067cba8...fac493f](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/067cba8229694e7fb9693313f51ca686746b620a?el=desc) will **decrease** coverage by `0.00%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   - Coverage   82.34%   82.34%   -0.01%     
   ==========================================
     Files         452      453       +1     
     Lines       54016    54060      +44     
   ==========================================
   + Hits        44481    44515      +34     
   - Misses       9535     9545      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [...dks/python/apache\_beam/runners/pipeline\_context.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9waXBlbGluZV9jb250ZXh0LnB5) | `92.80% <0.00%> (-1.01%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.90% <0.00%> (-0.76%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.40%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_dataframe.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RhdGFmcmFtZS5weQ==) | `91.66% <0.00%> (ø)` | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.32%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [067cba8...fac493f](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12888: [BEAM-10861]Moves PubSub Runner API encoding to Read/Write transforms

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12888:
URL: https://github.com/apache/beam/pull/12888#issuecomment-696562704


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=h1) Report
   > Merging [#12888](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2ff8337aa943623dfaabaf8b42876b4d9ee1c963?el=desc) will **increase** coverage by `0.01%`.
   > The diff coverage is `93.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12888/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12888      +/-   ##
   ==========================================
   + Coverage   82.32%   82.34%   +0.01%     
   ==========================================
     Files         454      453       -1     
     Lines       54088    54060      -28     
   ==========================================
   - Hits        44528    44515      -13     
   + Misses       9560     9545      -15     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.30% <50.00%> (-1.28%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `85.06% <96.29%> (+0.73%)` | :arrow_up: |
   | [.../python/apache\_beam/io/gcp/bigquery\_io\_metadata.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2lvX21ldGFkYXRhLnB5) | `86.95% <0.00%> (-3.67%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.96% <0.00%> (-0.17%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `79.78% <0.00%> (-0.01%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/io/external/snowflake.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZXh0ZXJuYWwvc25vd2ZsYWtlLnB5) | | |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (+0.17%)` | :arrow_up: |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `77.51% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `89.20% <0.00%> (+0.44%)` | :arrow_up: |
   | ... and [1 more](https://codecov.io/gh/apache/beam/pull/12888/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=footer). Last update [89c3f51...98defc3](https://codecov.io/gh/apache/beam/pull/12888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org