You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/12/30 14:59:14 UTC

[GitHub] [beam] ramazan-yapparov opened a new pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

ramazan-yapparov opened a new pull request #13636:
URL: https://github.com/apache/beam/pull/13636


   [Proposal] Apache Beam pipeline example to ingest data from Apache Kafka to Google Cloud Pub/Sub. It can be used as a Dataflow Flex template in the Google Cloud Platform.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [x] Update `CHANGES.md` with noteworthy changes.
    - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam
 .apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.a
 pache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam
 .apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website | Whitespace | Typescript
   --- | --- | --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/be
 am_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (46f44c9) into [master](https://codecov.io/gh/apache/beam/commit/21ffad3ea0a398376bfa17b746a38db0df97a84a?el=desc) (21ffad3) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.75%           
   =======================================
     Files         466      466           
     Lines       57525    57525           
   =======================================
   + Hits        47601    47606    +5     
   + Misses       9924     9919    -5     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.57% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.83%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...24ddb6e](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (0020886) into [master](https://codecov.io/gh/apache/beam/commit/6432274f8ef9aaa3ff1101451d51c8e260558f6b?el=desc) (6432274) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.74%           
   =======================================
     Files         466      466           
     Lines       57525    57520    -5     
   =======================================
   - Hits        47597    47593    -4     
   + Misses       9928     9927    -1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/transforms/external.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9leHRlcm5hbC5weQ==) | `71.70% <0.00%> (-0.39%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.57% <0.00%> (+0.10%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...f68fd02](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r563605230



##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/Utils.java
##########
@@ -162,4 +165,11 @@ public static boolean isSslSpecified(KafkaToPubsubOptions options) {
         || options.getKeystorePath() != null
         || options.getKeyPassword() != null;
   }
+
+  public static Map<String, Object> parseKafkaConsumerConfig(String kafkaConsumerConfig) {
+    return Arrays.stream(kafkaConsumerConfig.split(";"))
+        .map(s -> s.split("="))
+        .map(kv -> Pair.of(kv[0], kv[1]))
+        .collect(Collectors.toMap(Pair::getKey, Pair::getValue));

Review comment:
       Removed `Pair` related code, but left method in place.
   This class is created for such methods, in my opinion it is better to declare them in `Utils` in order to leave main class as clean as possible




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57527       +2     
   ==========================================
   + Hits        47601    47602       +1     
   - Misses       9924     9925       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.46% <0.00%> (-0.42%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (c66262f) into [master](https://codecov.io/gh/apache/beam/commit/6c9da02d553cc0d39d29b0293c9b37ac0ceec73e?el=desc) (6c9da02) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.76%   82.74%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57520    57520              
   ==========================================
   - Hits        47605    47597       -8     
   - Misses       9915     9923       +8     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.29% <0.00%> (-0.53%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [6c9da02...0020886](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r562275044



##########
File path: examples/java/build.gradle
##########
@@ -56,6 +56,7 @@ dependencies {
   compile library.java.vendored_guava_26_0_jre
   compile library.java.kafka_clients
   compile project(path: ":sdks:java:core", configuration: "shadow")
+  compile project(path: ":runners:direct-java", configuration: "shadow")

Review comment:
       Took a closer look at this. It looks like the dependency is necessary because `DirectOptions` is referenced in the new test. Could you make it a `testCompile` dependency?

##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/Utils.java
##########
@@ -162,4 +165,11 @@ public static boolean isSslSpecified(KafkaToPubsubOptions options) {
         || options.getKeystorePath() != null
         || options.getKeyPassword() != null;
   }
+
+  public static Map<String, Object> parseKafkaConsumerConfig(String kafkaConsumerConfig) {
+    return Arrays.stream(kafkaConsumerConfig.split(";"))
+        .map(s -> s.split("="))
+        .map(kv -> Pair.of(kv[0], kv[1]))
+        .collect(Collectors.toMap(Pair::getKey, Pair::getValue));

Review comment:
       nit: If I were writing this I would probably combine these two lines and avoid creating the `Pair`:
   ```suggestion
           .collect(Collectors.toMap(kv -> kv[0], kv -> kv[1]));
   ```
   If you prefer it with the Pair that's fine too.
   
   One thing I think we _should_ change is make this a private static method in `KafkaToPubsub`, since it's only user there.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (24ddb6e) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.75%           
   =======================================
     Files         466      466           
     Lines       57525    57525           
   =======================================
   + Hits        47601    47603    +2     
   + Misses       9924     9922    -2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `94.11% <0.00%> (+2.20%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (46f44c9) into [master](https://codecov.io/gh/apache/beam/commit/21ffad3ea0a398376bfa17b746a38db0df97a84a?el=desc) (21ffad3) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.75%           
   =======================================
     Files         466      466           
     Lines       57525    57525           
   =======================================
   + Hits        47601    47606    +5     
   + Misses       9924     9919    -5     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.57% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.83%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...46f44c9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (c66262f) into [master](https://codecov.io/gh/apache/beam/commit/6c9da02d553cc0d39d29b0293c9b37ac0ceec73e?el=desc) (6c9da02) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.76%   82.74%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57520    57520              
   ==========================================
   - Hits        47605    47597       -8     
   - Misses       9915     9923       +8     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.29% <0.00%> (-0.53%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [6c9da02...c66262f](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57527       +2     
   ==========================================
   + Hits        47601    47602       +1     
   - Misses       9924     9925       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.46% <0.00%> (-0.42%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (0020886) into [master](https://codecov.io/gh/apache/beam/commit/6c9da02d553cc0d39d29b0293c9b37ac0ceec73e?el=desc) (6c9da02) will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.76%   82.74%   -0.03%     
   ==========================================
     Files         466      466              
     Lines       57520    57520              
   ==========================================
   - Hits        47605    47593      -12     
   - Misses       9915     9927      +12     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.29% <0.00%> (-0.53%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.57% <0.00%> (-0.42%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [6c9da02...0020886](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (46f44c9) into [master](https://codecov.io/gh/apache/beam/commit/21ffad3ea0a398376bfa17b746a38db0df97a84a?el=desc) (21ffad3) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.75%           
   =======================================
     Files         466      466           
     Lines       57525    57525           
   =======================================
   + Hits        47601    47606    +5     
   + Misses       9924     9919    -5     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.57% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.83%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...46f44c9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r562275044



##########
File path: examples/java/build.gradle
##########
@@ -56,6 +56,7 @@ dependencies {
   compile library.java.vendored_guava_26_0_jre
   compile library.java.kafka_clients
   compile project(path: ":sdks:java:core", configuration: "shadow")
+  compile project(path: ":runners:direct-java", configuration: "shadow")

Review comment:
       Took a closer look at this. It looks like the dependency is necessary because `DirectOptions` is referenced in the new test. Could you make it a `testCompile` dependency?

##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/Utils.java
##########
@@ -162,4 +165,11 @@ public static boolean isSslSpecified(KafkaToPubsubOptions options) {
         || options.getKeystorePath() != null
         || options.getKeyPassword() != null;
   }
+
+  public static Map<String, Object> parseKafkaConsumerConfig(String kafkaConsumerConfig) {
+    return Arrays.stream(kafkaConsumerConfig.split(";"))
+        .map(s -> s.split("="))
+        .map(kv -> Pair.of(kv[0], kv[1]))
+        .collect(Collectors.toMap(Pair::getKey, Pair::getValue));

Review comment:
       nit: If I were writing this I would probably combine these two lines and avoid creating the `Pair`:
   ```suggestion
           .collect(Collectors.toMap(kv -> kv[0], kv -> kv[1]));
   ```
   If you prefer it with the Pair that's fine too.
   
   One thing I think we _should_ change is make this a private static method in `KafkaToPubsub`, since it's only user there.

##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/Utils.java
##########
@@ -162,4 +165,11 @@ public static boolean isSslSpecified(KafkaToPubsubOptions options) {
         || options.getKeystorePath() != null
         || options.getKeyPassword() != null;
   }
+
+  public static Map<String, Object> parseKafkaConsumerConfig(String kafkaConsumerConfig) {
+    return Arrays.stream(kafkaConsumerConfig.split(";"))
+        .map(s -> s.split("="))
+        .map(kv -> Pair.of(kv[0], kv[1]))
+        .collect(Collectors.toMap(Pair::getKey, Pair::getValue));

Review comment:
       nit: If I were writing this I would probably combine these two lines and avoid creating the `Pair`:
   ```suggestion
           .collect(Collectors.toMap(kv -> kv[0], kv -> kv[1]));
   ```
   If you prefer it with the Pair that's fine too.
   
   One thing I think we _should_ change is make this a private static method in `KafkaToPubsub`, since it's only used there.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47646      -10     
   - Misses       9902     9912      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-766967692


   Merged! Thanks a lot for sticking with this @ramazan-yapparov :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-766967692


   Merged! Thanks a lot for sticking with this @ramazan-yapparov :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47646      -10     
   - Misses       9902     9912      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (c66262f) into [master](https://codecov.io/gh/apache/beam/commit/6c9da02d553cc0d39d29b0293c9b37ac0ceec73e?el=desc) (6c9da02) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.76%   82.74%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57520    57520              
   ==========================================
   - Hits        47605    47597       -8     
   - Misses       9915     9923       +8     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.29% <0.00%> (-0.53%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [6c9da02...c66262f](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aaltay commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

aaltay commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-761285224


   > Hi @aaltay we tried to turn this test into IT but ran into some troubles, would be great if you could help us with them:
   > 
   > 1. When running the test all Docker containers are starting in the machine that is running the test,
   >    so in order for this test to work properly dataflow job should be able to reach test-runner machine by a public IP.
   >    I certainly can't do it on my local machine, not sure how it will behave when running in CI environment.
   > 2. When we pass our fake PubSub url into the dataflow job we receive following error, not sure how this can be avoided, looks like the job will only accept the real Cloud PubSub url:
   > 
   > ```json
   > {
   >   "code" : 400,
   >   "errors" : [ {
   >     "domain" : "global",
   >     "message" : "(f214233f9dbe6968): The workflow could not be created. Causes: (f214233f9dbe6719): http://localhost:49169 is not a valid Pub/Sub URL.",
   >     "reason" : "badRequest"
   >   } ],
   >   "message" : "(f214233f9dbe6968): The workflow could not be created. Causes: (f214233f9dbe6719): http://localhost:49169 is not a valid Pub/Sub URL.",
   >   "status" : "INVALID_ARGUMENT"
   > }
   > ```
   > 
   > FYI @TheNeuralBit
   
   I do not know the answer to these questions, and you can try asking on the dev@ list for a different answer.
   
   However, this does not sounds like a feasible test plan. Because, we do not generally use our test workers to mock services, and we generally do not want jenkins workers to be publicly reachable. A better approach, as part of the test setup, creating a new pubsub topic, publishing test messages there, let the pipeline do its job and at the end of the pipeline check that pipeline produced expected outputs.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.78%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47649       -7     
   - Misses       9902     9909       +7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit merged pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit merged pull request #13636:
URL: https://github.com/apache/beam/pull/13636


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (c66262f) into [master](https://codecov.io/gh/apache/beam/commit/6c9da02d553cc0d39d29b0293c9b37ac0ceec73e?el=desc) (6c9da02) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.76%   82.74%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57520    57520              
   ==========================================
   - Hits        47605    47597       -8     
   - Misses       9915     9923       +8     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.29% <0.00%> (-0.53%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [6c9da02...c66262f](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47645      -11     
   - Misses       9902     9913      +11     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (f68fd02) into [master](https://codecov.io/gh/apache/beam/commit/6432274f8ef9aaa3ff1101451d51c8e260558f6b?el=desc) (6432274) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.74%           
   =======================================
     Files         466      466           
     Lines       57525    57525           
   =======================================
   + Hits        47597    47598    +1     
   + Misses       9928     9927    -1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.44% <0.00%> (-0.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.41%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...f68fd02](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57527       +2     
   ==========================================
   + Hits        47601    47602       +1     
   - Misses       9924     9925       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.46% <0.00%> (-0.42%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.78%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47649       -7     
   - Misses       9902     9909       +7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/24179c3b39bf40db41d58547fff4054dde8ea24a?el=desc) (24179c3) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.75%   82.75%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57527    57527              
   ==========================================
   - Hits        47607    47606       -1     
   - Misses       9920     9921       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `89.92% <0.00%> (-0.36%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57527       +2     
   ==========================================
   + Hits        47601    47602       +1     
   - Misses       9924     9925       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.46% <0.00%> (-0.42%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r556952137



##########
File path: examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubE2ETest.java
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.complete.kafkatopubsub;
+
+import static org.apache.beam.examples.complete.kafkatopubsub.transforms.FormatTransform.readFromKafka;
+
+import com.google.auth.Credentials;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Supplier;
+import org.apache.beam.examples.complete.kafkatopubsub.utils.RunKafkaContainer;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Values;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.testcontainers.containers.PubSubEmulatorContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** E2E test for {@link KafkaToPubsub} pipeline. */
+public class KafkaToPubsubE2ETest {
+
+  @Rule public final transient TestPipeline pipeline = TestPipeline.fromOptions(OPTIONS);
+  @Rule public transient TestPubsubSignal signal = TestPubsubSignal.fromOptions(OPTIONS);
+
+  private static final String PUBSUB_EMULATOR_IMAGE =
+      "gcr.io/google.com/cloudsdktool/cloud-sdk:316.0.0-emulators";
+  private static final String PUBSUB_MESSAGE = "test pubsub message";
+  private static final String PROJECT_ID = "try-kafka-pubsub";
+  private static final String TOPIC_NAME = "listen-to-kafka";
+  private static final PubsubClient.TopicPath TOPIC_PATH =
+      PubsubClient.topicPathFromName(PROJECT_ID, TOPIC_NAME);
+  private static final PipelineOptions OPTIONS = TestPipeline.testingPipelineOptions();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    Credentials credentials = NoopCredentialFactory.fromOptions(OPTIONS).getCredential();
+    OPTIONS.as(GcpOptions.class).setGcpCredential(credentials);
+    OPTIONS.as(GcpOptions.class).setProject(PROJECT_ID);
+    setupPubsubContainer(OPTIONS.as(PubsubOptions.class));
+    createPubsubTopicForTest(OPTIONS.as(PubsubOptions.class));
+  }
+
+  @Test
+  public void testKafkaToPubsubE2E() throws IOException {
+    pipeline.getOptions().as(DirectOptions.class).setBlockOnRun(false);
+
+    RunKafkaContainer rkc = new RunKafkaContainer(PUBSUB_MESSAGE);
+    String bootstrapServer = rkc.getBootstrapServer();
+    String[] kafkaTopicsList = new String[] {rkc.getTopicName()};
+
+    String pubsubTopicPath = TOPIC_PATH.getPath();
+
+    Map<String, Object> kafkaConfig = new HashMap<>();
+    Map<String, String> sslConfig = new HashMap<>();
+
+    PCollection<KV<String, String>> readStrings =
+        pipeline.apply(
+            "readFromKafka",
+            readFromKafka(bootstrapServer, Arrays.asList(kafkaTopicsList), kafkaConfig, sslConfig));
+
+    PCollection<String> readFromPubsub =
+        readStrings
+            .apply(Values.create())
+            .apply("writeToPubSub", PubsubIO.writeStrings().to(pubsubTopicPath))
+            .getPipeline()
+            .apply("readFromPubsub", PubsubIO.readStrings().fromTopic(pubsubTopicPath));
+
+    readFromPubsub.apply(
+        "waitForTestMessage",
+        signal.signalSuccessWhen(
+            readFromPubsub.getCoder(),
+            input -> {
+              if (input == null) {
+                return false;
+              }
+              return input.stream().anyMatch(message -> Objects.equals(message, PUBSUB_MESSAGE));
+            }));
+
+    Supplier<Void> start = signal.waitForStart(Duration.standardSeconds(10));
+    pipeline.apply(signal.signalStart());
+    PipelineResult job = pipeline.run();
+    start.get();
+    signal.waitForSuccess(Duration.standardMinutes(2));
+    try {
+      job.cancel();
+    } catch (IOException | UnsupportedOperationException e) {
+      throw new AssertionError("Could not stop pipeline.", e);
+    }
+  }
+
+  private static void setupPubsubContainer(PubsubOptions options) {
+    PubSubEmulatorContainer emulator =
+        new PubSubEmulatorContainer(DockerImageName.parse(PUBSUB_EMULATOR_IMAGE));
+    emulator.start();
+    String pubsubUrl = emulator.getEmulatorEndpoint();
+    options.setPubsubRootUrl("http://" + pubsubUrl);
+  }
+
+  private static void createPubsubTopicForTest(PubsubOptions options) {
+    try {
+      PubsubClient pubsubClient = PubsubJsonClient.FACTORY.newClient(null, null, options);
+      pubsubClient.createTopic(TOPIC_PATH);
+    } catch (Exception e) {
+      throw new RuntimeException(e);

Review comment:
       Awesome! Thanks for making TestPubSub work with the emulator container :+1:
   
   Is the trick to making sure the test container is run before `TestPubsub` to make the container a `@ClassRule`?
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57527       +2     
   ==========================================
   + Hits        47601    47602       +1     
   - Misses       9924     9925       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.46% <0.00%> (-0.42%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r556955692



##########
File path: examples/java/build.gradle
##########
@@ -56,6 +56,7 @@ dependencies {
   compile library.java.vendored_guava_26_0_jre
   compile library.java.kafka_clients
   compile project(path: ":sdks:java:core", configuration: "shadow")
+  compile project(path: ":runners:direct-java", configuration: "shadow")

Review comment:
       Can we do without this?

##########
File path: examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubE2ETest.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.complete.kafkatopubsub;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasProperty;
+
+import com.google.auth.Credentials;
+import java.nio.charset.StandardCharsets;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import org.apache.beam.examples.complete.kafkatopubsub.options.KafkaToPubsubOptions;
+import org.apache.beam.examples.complete.kafkatopubsub.transforms.FormatTransform.FORMAT;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.TestPubsub;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.joda.time.Duration;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.PubSubEmulatorContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** E2E test for {@link KafkaToPubsub} pipeline. */
+public class KafkaToPubsubE2ETest {
+  private static final String PUBSUB_EMULATOR_IMAGE =
+      "gcr.io/google.com/cloudsdktool/cloud-sdk:316.0.0-emulators";
+  private static final String KAFKA_IMAGE_NAME = "confluentinc/cp-kafka:5.4.3";
+  private static final String PUBSUB_MESSAGE = "test pubsub message";
+  private static final String KAFKA_TOPIC_NAME = "messages-topic";
+  private static final String PROJECT_ID = "try-kafka-pubsub";
+  private static final PipelineOptions OPTIONS = TestPipeline.testingPipelineOptions();
+
+  @ClassRule
+  public static final PubSubEmulatorContainer PUB_SUB_EMULATOR_CONTAINER =
+      new PubSubEmulatorContainer(DockerImageName.parse(PUBSUB_EMULATOR_IMAGE));
+
+  @ClassRule
+  public static final KafkaContainer KAFKA_CONTAINER =
+      new KafkaContainer(DockerImageName.parse(KAFKA_IMAGE_NAME));
+
+  @Rule public final transient TestPipeline pipeline = TestPipeline.fromOptions(OPTIONS);
+  @Rule public final transient TestPubsub testPubsub = TestPubsub.fromOptions(OPTIONS);
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    Credentials credentials = NoopCredentialFactory.fromOptions(OPTIONS).getCredential();
+    OPTIONS.as(DirectOptions.class).setBlockOnRun(false);
+    OPTIONS.as(GcpOptions.class).setGcpCredential(credentials);
+    OPTIONS.as(GcpOptions.class).setProject(PROJECT_ID);
+    OPTIONS
+        .as(PubsubOptions.class)
+        .setPubsubRootUrl("http://" + PUB_SUB_EMULATOR_CONTAINER.getEmulatorEndpoint());
+    OPTIONS.as(KafkaToPubsubOptions.class).setOutputFormat(FORMAT.PUBSUB);
+    OPTIONS
+        .as(KafkaToPubsubOptions.class)
+        .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers());
+    OPTIONS.as(KafkaToPubsubOptions.class).setInputTopics(KAFKA_TOPIC_NAME);
+    OPTIONS
+        .as(KafkaToPubsubOptions.class)
+        .setKafkaConsumerConfig(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG + "=earliest");
+  }
+
+  @Before
+  public void setUp() {
+    OPTIONS.as(KafkaToPubsubOptions.class).setOutputTopic(testPubsub.topicPath().getPath());
+  }
+
+  @Test
+  public void testKafkaToPubsubE2E() throws Exception {
+    PipelineResult job = KafkaToPubsub.run(pipeline, OPTIONS.as(KafkaToPubsubOptions.class));
+
+    sendKafkaMessage();
+    testPubsub
+        .assertThatTopicEventuallyReceives(
+            hasProperty("payload", equalTo(PUBSUB_MESSAGE.getBytes(StandardCharsets.UTF_8))))
+        .waitForUpTo(Duration.standardMinutes(1));

Review comment:
       We have a precommit check that should run this example on Dataflow. It looks like this test isn't run there [now](https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/12132/testReport/org.apache.beam.examples/), but that's likely because the name is Test and not IT. 
   
   When you rename the test and this runs on Dataflow, this timeout won't be long enough, since Dataflow takes several minutes to start up workers. You should bump it up to 10 minutes probably.

##########
File path: examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubE2ETest.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.complete.kafkatopubsub;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasProperty;
+
+import com.google.auth.Credentials;
+import java.nio.charset.StandardCharsets;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import org.apache.beam.examples.complete.kafkatopubsub.options.KafkaToPubsubOptions;
+import org.apache.beam.examples.complete.kafkatopubsub.transforms.FormatTransform.FORMAT;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.TestPubsub;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.joda.time.Duration;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.PubSubEmulatorContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** E2E test for {@link KafkaToPubsub} pipeline. */
+public class KafkaToPubsubE2ETest {

Review comment:
       Could you rename this to `KafkaToPubsubIT`? Our build files assume `*Test` is a unit test and `*IT` is an integration test.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.75%           
   =======================================
     Files         466      466           
     Lines       57525    57527    +2     
   =======================================
   + Hits        47601    47606    +5     
   + Misses       9924     9921    -3     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r550277830



##########
File path: examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/utils/RunKafkaContainer.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.complete.kafkatopubsub.utils;
+
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** Run kafka container in separate thread to produce message. */
+public class RunKafkaContainer {
+
+  private static final String KAFKA_IMAGE_NAME = "confluentinc/cp-kafka:5.4.3";
+  private final String topicName;
+  private final KafkaProducer<String, String> producer;
+  private final String bootstrapServer;
+
+  public RunKafkaContainer(String pubsubMessage) {
+    bootstrapServer = setupKafkaContainer();
+    topicName = "messages-topic";
+    producer =
+        new KafkaProducer<>(
+            ImmutableMap.of(
+                ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
+                bootstrapServer,
+                ProducerConfig.CLIENT_ID_CONFIG,
+                UUID.randomUUID().toString()),
+            new StringSerializer(),
+            new StringSerializer());
+    Runnable kafkaProducer =
+        () -> {
+          try {
+            producer.send(new ProducerRecord<>(topicName, "testcontainers", pubsubMessage)).get();
+            System.out.println("Producer sent");
+          } catch (ExecutionException | InterruptedException e) {
+            throw new RuntimeException("Something went wrong in kafka producer", e);
+          }
+        };
+    // Without saving `.schedule(...)` result to variable checkframework will fail
+    @SuppressWarnings("unused")
+    ScheduledFuture<?> schedule =
+        Executors.newSingleThreadScheduledExecutor().schedule(kafkaProducer, 10, TimeUnit.SECONDS);

Review comment:
       Could you instead inject the data in the test after the pipeline has started?

##########
File path: examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubE2ETest.java
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.complete.kafkatopubsub;
+
+import static org.apache.beam.examples.complete.kafkatopubsub.transforms.FormatTransform.readFromKafka;
+
+import com.google.auth.Credentials;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Supplier;
+import org.apache.beam.examples.complete.kafkatopubsub.utils.RunKafkaContainer;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Values;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.testcontainers.containers.PubSubEmulatorContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** E2E test for {@link KafkaToPubsub} pipeline. */
+public class KafkaToPubsubE2ETest {
+
+  @Rule public final transient TestPipeline pipeline = TestPipeline.fromOptions(OPTIONS);
+  @Rule public transient TestPubsubSignal signal = TestPubsubSignal.fromOptions(OPTIONS);
+
+  private static final String PUBSUB_EMULATOR_IMAGE =
+      "gcr.io/google.com/cloudsdktool/cloud-sdk:316.0.0-emulators";
+  private static final String PUBSUB_MESSAGE = "test pubsub message";
+  private static final String PROJECT_ID = "try-kafka-pubsub";
+  private static final String TOPIC_NAME = "listen-to-kafka";
+  private static final PubsubClient.TopicPath TOPIC_PATH =
+      PubsubClient.topicPathFromName(PROJECT_ID, TOPIC_NAME);
+  private static final PipelineOptions OPTIONS = TestPipeline.testingPipelineOptions();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    Credentials credentials = NoopCredentialFactory.fromOptions(OPTIONS).getCredential();
+    OPTIONS.as(GcpOptions.class).setGcpCredential(credentials);
+    OPTIONS.as(GcpOptions.class).setProject(PROJECT_ID);
+    setupPubsubContainer(OPTIONS.as(PubsubOptions.class));
+    createPubsubTopicForTest(OPTIONS.as(PubsubOptions.class));
+  }
+
+  @Test
+  public void testKafkaToPubsubE2E() throws IOException {
+    pipeline.getOptions().as(DirectOptions.class).setBlockOnRun(false);
+
+    RunKafkaContainer rkc = new RunKafkaContainer(PUBSUB_MESSAGE);
+    String bootstrapServer = rkc.getBootstrapServer();
+    String[] kafkaTopicsList = new String[] {rkc.getTopicName()};
+
+    String pubsubTopicPath = TOPIC_PATH.getPath();
+
+    Map<String, Object> kafkaConfig = new HashMap<>();
+    Map<String, String> sslConfig = new HashMap<>();
+
+    PCollection<KV<String, String>> readStrings =
+        pipeline.apply(
+            "readFromKafka",
+            readFromKafka(bootstrapServer, Arrays.asList(kafkaTopicsList), kafkaConfig, sslConfig));
+
+    PCollection<String> readFromPubsub =
+        readStrings
+            .apply(Values.create())
+            .apply("writeToPubSub", PubsubIO.writeStrings().to(pubsubTopicPath))
+            .getPipeline()
+            .apply("readFromPubsub", PubsubIO.readStrings().fromTopic(pubsubTopicPath));
+
+    readFromPubsub.apply(
+        "waitForTestMessage",
+        signal.signalSuccessWhen(
+            readFromPubsub.getCoder(),
+            input -> {
+              if (input == null) {
+                return false;
+              }
+              return input.stream().anyMatch(message -> Objects.equals(message, PUBSUB_MESSAGE));
+            }));
+
+    Supplier<Void> start = signal.waitForStart(Duration.standardSeconds(10));
+    pipeline.apply(signal.signalStart());
+    PipelineResult job = pipeline.run();
+    start.get();
+    signal.waitForSuccess(Duration.standardMinutes(2));
+    try {
+      job.cancel();
+    } catch (IOException | UnsupportedOperationException e) {
+      throw new AssertionError("Could not stop pipeline.", e);
+    }
+  }
+
+  private static void setupPubsubContainer(PubsubOptions options) {
+    PubSubEmulatorContainer emulator =
+        new PubSubEmulatorContainer(DockerImageName.parse(PUBSUB_EMULATOR_IMAGE));
+    emulator.start();
+    String pubsubUrl = emulator.getEmulatorEndpoint();
+    options.setPubsubRootUrl("http://" + pubsubUrl);
+  }
+
+  private static void createPubsubTopicForTest(PubsubOptions options) {
+    try {
+      PubsubClient pubsubClient = PubsubJsonClient.FACTORY.newClient(null, null, options);
+      pubsubClient.createTopic(TOPIC_PATH);
+    } catch (Exception e) {
+      throw new RuntimeException(e);

Review comment:
       You might look at using `TestPubsub` to create the test topic instead of creating it manually. `TestPubsub` also has a method that you can use to check the topic receives some expected messages, which would save you from creating the `readFromPubsub` transform to signal success from within the pipeline: https://github.com/apache/beam/blob/5e17b69630cc61b42523582b6bb94cb8191dcef5/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java#L342
   
   It will be tricky to make this work with the pubsub test container though, since we'll need to start the test container _before_ the TestPubsub Rule initializes its topic. This would be really useful infrastructure though as it would allow us to run many other pubsub tests against the fake instead of prod pubsub.

##########
File path: examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubE2ETest.java
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.complete.kafkatopubsub;
+
+import static org.apache.beam.examples.complete.kafkatopubsub.transforms.FormatTransform.readFromKafka;
+
+import com.google.auth.Credentials;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Supplier;
+import org.apache.beam.examples.complete.kafkatopubsub.utils.RunKafkaContainer;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Values;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.testcontainers.containers.PubSubEmulatorContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** E2E test for {@link KafkaToPubsub} pipeline. */
+public class KafkaToPubsubE2ETest {
+
+  @Rule public final transient TestPipeline pipeline = TestPipeline.fromOptions(OPTIONS);
+  @Rule public transient TestPubsubSignal signal = TestPubsubSignal.fromOptions(OPTIONS);
+
+  private static final String PUBSUB_EMULATOR_IMAGE =
+      "gcr.io/google.com/cloudsdktool/cloud-sdk:316.0.0-emulators";
+  private static final String PUBSUB_MESSAGE = "test pubsub message";
+  private static final String PROJECT_ID = "try-kafka-pubsub";
+  private static final String TOPIC_NAME = "listen-to-kafka";
+  private static final PubsubClient.TopicPath TOPIC_PATH =
+      PubsubClient.topicPathFromName(PROJECT_ID, TOPIC_NAME);
+  private static final PipelineOptions OPTIONS = TestPipeline.testingPipelineOptions();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    Credentials credentials = NoopCredentialFactory.fromOptions(OPTIONS).getCredential();
+    OPTIONS.as(GcpOptions.class).setGcpCredential(credentials);
+    OPTIONS.as(GcpOptions.class).setProject(PROJECT_ID);
+    setupPubsubContainer(OPTIONS.as(PubsubOptions.class));
+    createPubsubTopicForTest(OPTIONS.as(PubsubOptions.class));
+  }
+
+  @Test
+  public void testKafkaToPubsubE2E() throws IOException {
+    pipeline.getOptions().as(DirectOptions.class).setBlockOnRun(false);
+
+    RunKafkaContainer rkc = new RunKafkaContainer(PUBSUB_MESSAGE);
+    String bootstrapServer = rkc.getBootstrapServer();
+    String[] kafkaTopicsList = new String[] {rkc.getTopicName()};
+
+    String pubsubTopicPath = TOPIC_PATH.getPath();
+
+    Map<String, Object> kafkaConfig = new HashMap<>();
+    Map<String, String> sslConfig = new HashMap<>();
+
+    PCollection<KV<String, String>> readStrings =
+        pipeline.apply(
+            "readFromKafka",
+            readFromKafka(bootstrapServer, Arrays.asList(kafkaTopicsList), kafkaConfig, sslConfig));
+
+    PCollection<String> readFromPubsub =
+        readStrings
+            .apply(Values.create())
+            .apply("writeToPubSub", PubsubIO.writeStrings().to(pubsubTopicPath))
+            .getPipeline()
+            .apply("readFromPubsub", PubsubIO.readStrings().fromTopic(pubsubTopicPath));

Review comment:
       It would be preferable to run the exact KafkaToPubsub pipeline, then use utilities outside of the pipeline to inject data to the kafka topic, and then to verify the pubsub topic receives the expected messages. As noted in my other comment `TestPubsub` can help with the latter.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-764485252


   Thanks for clarification @TheNeuralBit 
   Yes, it is running with a DirectRunner in a Jenkins job.
   Here is the [commit](https://github.com/apache/beam/pull/13636/commits/46f44c96e545900289af0309fa238442d4e57d4b) that intentionally breaks the test
   And here is its [test failure report](https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/15512/testReport/junit/org.apache.beam.examples.complete.kafkatopubsub/KafkaToPubsubE2ETest/testKafkaToPubsubE2E/)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit merged pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit merged pull request #13636:
URL: https://github.com/apache/beam/pull/13636


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (0020886) into [master](https://codecov.io/gh/apache/beam/commit/6c9da02d553cc0d39d29b0293c9b37ac0ceec73e?el=desc) (6c9da02) will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.76%   82.74%   -0.03%     
   ==========================================
     Files         466      466              
     Lines       57520    57520              
   ==========================================
   - Hits        47605    47593      -12     
   - Misses       9915     9927      +12     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.29% <0.00%> (-0.53%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.57% <0.00%> (-0.42%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [6c9da02...0020886](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-766621593


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r557096068



##########
File path: examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubE2ETest.java
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.complete.kafkatopubsub;
+
+import static org.apache.beam.examples.complete.kafkatopubsub.transforms.FormatTransform.readFromKafka;
+
+import com.google.auth.Credentials;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Supplier;
+import org.apache.beam.examples.complete.kafkatopubsub.utils.RunKafkaContainer;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Values;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.testcontainers.containers.PubSubEmulatorContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** E2E test for {@link KafkaToPubsub} pipeline. */
+public class KafkaToPubsubE2ETest {
+
+  @Rule public final transient TestPipeline pipeline = TestPipeline.fromOptions(OPTIONS);
+  @Rule public transient TestPubsubSignal signal = TestPubsubSignal.fromOptions(OPTIONS);
+
+  private static final String PUBSUB_EMULATOR_IMAGE =
+      "gcr.io/google.com/cloudsdktool/cloud-sdk:316.0.0-emulators";
+  private static final String PUBSUB_MESSAGE = "test pubsub message";
+  private static final String PROJECT_ID = "try-kafka-pubsub";
+  private static final String TOPIC_NAME = "listen-to-kafka";
+  private static final PubsubClient.TopicPath TOPIC_PATH =
+      PubsubClient.topicPathFromName(PROJECT_ID, TOPIC_NAME);
+  private static final PipelineOptions OPTIONS = TestPipeline.testingPipelineOptions();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    Credentials credentials = NoopCredentialFactory.fromOptions(OPTIONS).getCredential();
+    OPTIONS.as(GcpOptions.class).setGcpCredential(credentials);
+    OPTIONS.as(GcpOptions.class).setProject(PROJECT_ID);
+    setupPubsubContainer(OPTIONS.as(PubsubOptions.class));
+    createPubsubTopicForTest(OPTIONS.as(PubsubOptions.class));
+  }
+
+  @Test
+  public void testKafkaToPubsubE2E() throws IOException {
+    pipeline.getOptions().as(DirectOptions.class).setBlockOnRun(false);
+
+    RunKafkaContainer rkc = new RunKafkaContainer(PUBSUB_MESSAGE);
+    String bootstrapServer = rkc.getBootstrapServer();
+    String[] kafkaTopicsList = new String[] {rkc.getTopicName()};
+
+    String pubsubTopicPath = TOPIC_PATH.getPath();
+
+    Map<String, Object> kafkaConfig = new HashMap<>();
+    Map<String, String> sslConfig = new HashMap<>();
+
+    PCollection<KV<String, String>> readStrings =
+        pipeline.apply(
+            "readFromKafka",
+            readFromKafka(bootstrapServer, Arrays.asList(kafkaTopicsList), kafkaConfig, sslConfig));
+
+    PCollection<String> readFromPubsub =
+        readStrings
+            .apply(Values.create())
+            .apply("writeToPubSub", PubsubIO.writeStrings().to(pubsubTopicPath))
+            .getPipeline()
+            .apply("readFromPubsub", PubsubIO.readStrings().fromTopic(pubsubTopicPath));
+
+    readFromPubsub.apply(
+        "waitForTestMessage",
+        signal.signalSuccessWhen(
+            readFromPubsub.getCoder(),
+            input -> {
+              if (input == null) {
+                return false;
+              }
+              return input.stream().anyMatch(message -> Objects.equals(message, PUBSUB_MESSAGE));
+            }));
+
+    Supplier<Void> start = signal.waitForStart(Duration.standardSeconds(10));
+    pipeline.apply(signal.signalStart());
+    PipelineResult job = pipeline.run();
+    start.get();
+    signal.waitForSuccess(Duration.standardMinutes(2));
+    try {
+      job.cancel();
+    } catch (IOException | UnsupportedOperationException e) {
+      throw new AssertionError("Could not stop pipeline.", e);
+    }
+  }
+
+  private static void setupPubsubContainer(PubsubOptions options) {
+    PubSubEmulatorContainer emulator =
+        new PubSubEmulatorContainer(DockerImageName.parse(PUBSUB_EMULATOR_IMAGE));
+    emulator.start();
+    String pubsubUrl = emulator.getEmulatorEndpoint();
+    options.setPubsubRootUrl("http://" + pubsubUrl);
+  }
+
+  private static void createPubsubTopicForTest(PubsubOptions options) {
+    try {
+      PubsubClient pubsubClient = PubsubJsonClient.FACTORY.newClient(null, null, options);
+      pubsubClient.createTopic(TOPIC_PATH);
+    } catch (Exception e) {
+      throw new RuntimeException(e);

Review comment:
       Exactly




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47645      -11     
   - Misses       9902     9913      +11     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (f68fd02) into [master](https://codecov.io/gh/apache/beam/commit/21ffad3ea0a398376bfa17b746a38db0df97a84a?el=desc) (21ffad3) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57525              
   ==========================================
   - Hits        47601    47598       -3     
   - Misses       9924     9927       +3     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.44% <0.00%> (-0.26%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.83%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...f68fd02](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (24ddb6e) into [master](https://codecov.io/gh/apache/beam/commit/21ffad3ea0a398376bfa17b746a38db0df97a84a?el=desc) (21ffad3) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.75%           
   =======================================
     Files         466      466           
     Lines       57525    57525           
   =======================================
   + Hits        47601    47603    +2     
   + Misses       9924     9922    -2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.83%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `94.11% <0.00%> (+2.20%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...24ddb6e](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57527       +2     
   ==========================================
   + Hits        47601    47602       +1     
   - Misses       9924     9925       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.46% <0.00%> (-0.42%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47646      -10     
   - Misses       9902     9912      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47646      -10     
   - Misses       9902     9912      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (24ddb6e) into [master](https://codecov.io/gh/apache/beam/commit/21ffad3ea0a398376bfa17b746a38db0df97a84a?el=desc) (21ffad3) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.75%           
   =======================================
     Files         466      466           
     Lines       57525    57525           
   =======================================
   + Hits        47601    47603    +2     
   + Misses       9924     9922    -2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.83%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `94.11% <0.00%> (+2.20%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...24ddb6e](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47645      -11     
   - Misses       9902     9913      +11     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-766706792


   Thanks, fixed everything


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-764485252


   Thanks for clarification @TheNeuralBit 
   Yes, it is running with a DirectRunner in a Jenkins job.
   Here is the [commit](https://github.com/apache/beam/pull/13636/commits/46f44c96e545900289af0309fa238442d4e57d4b) that intentionally breaks the test
   And here is its [test failure report](https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/15512/testReport/junit/org.apache.beam.examples.complete.kafkatopubsub/KafkaToPubsubE2ETest/testKafkaToPubsubE2E/)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57527       +2     
   ==========================================
   + Hits        47601    47602       +1     
   - Misses       9924     9925       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.46% <0.00%> (-0.42%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57527       +2     
   ==========================================
   + Hits        47601    47602       +1     
   - Misses       9924     9925       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.46% <0.00%> (-0.42%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (f68fd02) into [master](https://codecov.io/gh/apache/beam/commit/21ffad3ea0a398376bfa17b746a38db0df97a84a?el=desc) (21ffad3) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57525              
   ==========================================
   - Hits        47601    47598       -3     
   - Misses       9924     9927       +3     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.44% <0.00%> (-0.26%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.83%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...f68fd02](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (0020886) into [master](https://codecov.io/gh/apache/beam/commit/6c9da02d553cc0d39d29b0293c9b37ac0ceec73e?el=desc) (6c9da02) will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.76%   82.74%   -0.03%     
   ==========================================
     Files         466      466              
     Lines       57520    57520              
   ==========================================
   - Hits        47605    47593      -12     
   - Misses       9915     9927      +12     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.29% <0.00%> (-0.53%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.57% <0.00%> (-0.42%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [6c9da02...0020886](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (5e17b69) into [master](https://codecov.io/gh/apache/beam/commit/6c9da02d553cc0d39d29b0293c9b37ac0ceec73e?el=desc) (6c9da02) will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.76%   82.73%   -0.03%     
   ==========================================
     Files         466      466              
     Lines       57520    57514       -6     
   ==========================================
   - Hits        47605    47584      -21     
   - Misses       9915     9930      +15     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
   | [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `90.32% <0.00%> (-0.79%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.32% <0.00%> (-0.67%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.29% <0.00%> (-0.53%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/ptransform.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wdHJhbnNmb3JtLnB5) | `92.96% <0.00%> (-0.11%)` | :arrow_down: |
   | [setup.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
   | [...ache\_beam/examples/cookbook/bigquery\_side\_input.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfc2lkZV9pbnB1dC5weQ==) | `56.14% <0.00%> (ø)` | |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   | ... and [4 more](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [6c9da02...c66262f](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (f68fd02) into [master](https://codecov.io/gh/apache/beam/commit/21ffad3ea0a398376bfa17b746a38db0df97a84a?el=desc) (21ffad3) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57525              
   ==========================================
   - Hits        47601    47598       -3     
   - Misses       9924     9927       +3     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.44% <0.00%> (-0.26%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.83%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...46f44c9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47645      -11     
   - Misses       9902     9913      +11     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57527       +2     
   ==========================================
   + Hits        47601    47602       +1     
   - Misses       9924     9925       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.46% <0.00%> (-0.42%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-765724755


   Unfortunately it looks like we got a merge conflict with https://github.com/apache/beam/pull/13649 could you merge or rebase to resolve it?
   
   I thought I had permission to edit your branch so I tried to fix it myself (the conflict is trivial), but I wasn't allowed to push the merge commit :/


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (c66262f) into [master](https://codecov.io/gh/apache/beam/commit/6c9da02d553cc0d39d29b0293c9b37ac0ceec73e?el=desc) (6c9da02) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.76%   82.74%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57520    57520              
   ==========================================
   - Hits        47605    47597       -8     
   - Misses       9915     9923       +8     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.29% <0.00%> (-0.53%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [6c9da02...c66262f](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47645      -11     
   - Misses       9902     9913      +11     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47645      -11     
   - Misses       9902     9913      +11     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (46f44c9) into [master](https://codecov.io/gh/apache/beam/commit/21ffad3ea0a398376bfa17b746a38db0df97a84a?el=desc) (21ffad3) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.75%           
   =======================================
     Files         466      466           
     Lines       57525    57525           
   =======================================
   + Hits        47601    47606    +5     
   + Misses       9924     9919    -5     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.57% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.83%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...46f44c9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (5e17b69) into [master](https://codecov.io/gh/apache/beam/commit/b6243e7711c34735973491393a920fa358ae379d?el=desc) (b6243e7) will **decrease** coverage by `0.01%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.73%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57518    57514       -4     
   ==========================================
   - Hits        47594    47584      -10     
   - Misses       9924     9930       +6     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `90.47% <ø> (ø)` | |
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `92.19% <100.00%> (-0.15%)` | :arrow_down: |
   | [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.85% <100.00%> (ø)` | |
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [setup.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [e9a7b44...bc7b38a](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47646      -10     
   - Misses       9902     9912      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47646      -10     
   - Misses       9902     9912      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-766621593






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-766673695


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47646      -10     
   - Misses       9902     9912      +10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-763875967


   Sorry for the confusion @aaltay and @ramazan-yapparov. I responded on the dev@ thread. I think what you had before, `*Test`, was the right thing to do. Alternatively, you could make the test use prod pubsub (instead of the test container) when it's running with any runner other than the DirectRunner. Either option is fine with me.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r563605230



##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/Utils.java
##########
@@ -162,4 +165,11 @@ public static boolean isSslSpecified(KafkaToPubsubOptions options) {
         || options.getKeystorePath() != null
         || options.getKeyPassword() != null;
   }
+
+  public static Map<String, Object> parseKafkaConsumerConfig(String kafkaConsumerConfig) {
+    return Arrays.stream(kafkaConsumerConfig.split(";"))
+        .map(s -> s.split("="))
+        .map(kv -> Pair.of(kv[0], kv[1]))
+        .collect(Collectors.toMap(Pair::getKey, Pair::getValue));

Review comment:
       Removed `Pair` related code, but left method in place.
   This class is created for such methods, in my opinion it is better to declare them in `Utils` in order to leave main class as clean as possible




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (0020886) into [master](https://codecov.io/gh/apache/beam/commit/6c9da02d553cc0d39d29b0293c9b37ac0ceec73e?el=desc) (6c9da02) will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.76%   82.74%   -0.03%     
   ==========================================
     Files         466      466              
     Lines       57520    57520              
   ==========================================
   - Hits        47605    47593      -12     
   - Misses       9915     9927      +12     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.29% <0.00%> (-0.53%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.57% <0.00%> (-0.42%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [6c9da02...0020886](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57527       +2     
   ==========================================
   + Hits        47601    47602       +1     
   - Misses       9924     9925       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.46% <0.00%> (-0.42%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r556481572



##########
File path: examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/utils/RunKafkaContainer.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.complete.kafkatopubsub.utils;
+
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** Run kafka container in separate thread to produce message. */
+public class RunKafkaContainer {
+
+  private static final String KAFKA_IMAGE_NAME = "confluentinc/cp-kafka:5.4.3";
+  private final String topicName;
+  private final KafkaProducer<String, String> producer;
+  private final String bootstrapServer;
+
+  public RunKafkaContainer(String pubsubMessage) {
+    bootstrapServer = setupKafkaContainer();
+    topicName = "messages-topic";
+    producer =
+        new KafkaProducer<>(
+            ImmutableMap.of(
+                ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
+                bootstrapServer,
+                ProducerConfig.CLIENT_ID_CONFIG,
+                UUID.randomUUID().toString()),
+            new StringSerializer(),
+            new StringSerializer());
+    Runnable kafkaProducer =
+        () -> {
+          try {
+            producer.send(new ProducerRecord<>(topicName, "testcontainers", pubsubMessage)).get();
+            System.out.println("Producer sent");
+          } catch (ExecutionException | InterruptedException e) {
+            throw new RuntimeException("Something went wrong in kafka producer", e);
+          }
+        };
+    // Without saving `.schedule(...)` result to variable checkframework will fail
+    @SuppressWarnings("unused")
+    ScheduledFuture<?> schedule =
+        Executors.newSingleThreadScheduledExecutor().schedule(kafkaProducer, 10, TimeUnit.SECONDS);

Review comment:
       Fixed, thanks




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r556481994



##########
File path: examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubE2ETest.java
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.complete.kafkatopubsub;
+
+import static org.apache.beam.examples.complete.kafkatopubsub.transforms.FormatTransform.readFromKafka;
+
+import com.google.auth.Credentials;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Supplier;
+import org.apache.beam.examples.complete.kafkatopubsub.utils.RunKafkaContainer;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Values;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.testcontainers.containers.PubSubEmulatorContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** E2E test for {@link KafkaToPubsub} pipeline. */
+public class KafkaToPubsubE2ETest {
+
+  @Rule public final transient TestPipeline pipeline = TestPipeline.fromOptions(OPTIONS);
+  @Rule public transient TestPubsubSignal signal = TestPubsubSignal.fromOptions(OPTIONS);
+
+  private static final String PUBSUB_EMULATOR_IMAGE =
+      "gcr.io/google.com/cloudsdktool/cloud-sdk:316.0.0-emulators";
+  private static final String PUBSUB_MESSAGE = "test pubsub message";
+  private static final String PROJECT_ID = "try-kafka-pubsub";
+  private static final String TOPIC_NAME = "listen-to-kafka";
+  private static final PubsubClient.TopicPath TOPIC_PATH =
+      PubsubClient.topicPathFromName(PROJECT_ID, TOPIC_NAME);
+  private static final PipelineOptions OPTIONS = TestPipeline.testingPipelineOptions();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    Credentials credentials = NoopCredentialFactory.fromOptions(OPTIONS).getCredential();
+    OPTIONS.as(GcpOptions.class).setGcpCredential(credentials);
+    OPTIONS.as(GcpOptions.class).setProject(PROJECT_ID);
+    setupPubsubContainer(OPTIONS.as(PubsubOptions.class));
+    createPubsubTopicForTest(OPTIONS.as(PubsubOptions.class));
+  }
+
+  @Test
+  public void testKafkaToPubsubE2E() throws IOException {
+    pipeline.getOptions().as(DirectOptions.class).setBlockOnRun(false);
+
+    RunKafkaContainer rkc = new RunKafkaContainer(PUBSUB_MESSAGE);
+    String bootstrapServer = rkc.getBootstrapServer();
+    String[] kafkaTopicsList = new String[] {rkc.getTopicName()};
+
+    String pubsubTopicPath = TOPIC_PATH.getPath();
+
+    Map<String, Object> kafkaConfig = new HashMap<>();
+    Map<String, String> sslConfig = new HashMap<>();
+
+    PCollection<KV<String, String>> readStrings =
+        pipeline.apply(
+            "readFromKafka",
+            readFromKafka(bootstrapServer, Arrays.asList(kafkaTopicsList), kafkaConfig, sslConfig));
+
+    PCollection<String> readFromPubsub =
+        readStrings
+            .apply(Values.create())
+            .apply("writeToPubSub", PubsubIO.writeStrings().to(pubsubTopicPath))
+            .getPipeline()
+            .apply("readFromPubsub", PubsubIO.readStrings().fromTopic(pubsubTopicPath));

Review comment:
       Fixed that, now instead of building the pipeline in the test we execute `KafkaToPubsub#run` method




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.78%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47649       -7     
   - Misses       9902     9909       +7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (24ddb6e) into [master](https://codecov.io/gh/apache/beam/commit/21ffad3ea0a398376bfa17b746a38db0df97a84a?el=desc) (21ffad3) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.75%           
   =======================================
     Files         466      466           
     Lines       57525    57525           
   =======================================
   + Hits        47601    47603    +2     
   + Misses       9924     9922    -2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.83%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `94.11% <0.00%> (+2.20%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...24ddb6e](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47645      -11     
   - Misses       9902     9913      +11     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (24ddb6e) into [master](https://codecov.io/gh/apache/beam/commit/21ffad3ea0a398376bfa17b746a38db0df97a84a?el=desc) (21ffad3) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.75%           
   =======================================
     Files         466      466           
     Lines       57525    57525           
   =======================================
   + Hits        47601    47603    +2     
   + Misses       9924     9922    -2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.83%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `94.11% <0.00%> (+2.20%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...24ddb6e](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-765318308


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (9e6ac20) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.77%   -0.02%     
   ==========================================
     Files         466      466              
     Lines       57558    57558              
   ==========================================
   - Hits        47656    47645      -11     
   - Misses       9902     9913      +11     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/4861df410b19059fdc281f47ecf6c8f09c92e941?el=desc) (4861df4) will **decrease** coverage by `0.04%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.79%   82.75%   -0.05%     
   ==========================================
     Files         466      466              
     Lines       57558    57527      -31     
   ==========================================
   - Hits        47656    47606      -50     
   - Misses       9902     9921      +19     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...\_beam/runners/portability/sdk\_container\_builder.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zZGtfY29udGFpbmVyX2J1aWxkZXIucHk=) | `31.41% <0.00%> (-11.87%)` | :arrow_down: |
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-7.15%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `91.91% <0.00%> (-2.21%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.52% <0.00%> (-1.80%)` | :arrow_down: |
   | [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.66% <0.00%> (-0.18%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (-0.10%)` | :arrow_down: |
   | [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `94.60% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   | ... and [1 more](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [4861df4...9e6ac20](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (f68fd02) into [master](https://codecov.io/gh/apache/beam/commit/6432274f8ef9aaa3ff1101451d51c8e260558f6b?el=desc) (6432274) will **increase** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #13636   +/-   ##
   =======================================
     Coverage   82.74%   82.74%           
   =======================================
     Files         466      466           
     Lines       57525    57525           
   =======================================
   + Hits        47597    47598    +1     
   + Misses       9928     9927    -1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.44% <0.00%> (-0.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.88% <0.00%> (+0.41%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [21ffad3...f68fd02](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r562274474



##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/Utils.java
##########
@@ -162,4 +165,11 @@ public static boolean isSslSpecified(KafkaToPubsubOptions options) {
         || options.getKeystorePath() != null
         || options.getKeyPassword() != null;
   }
+
+  public static Map<String, Object> parseKafkaConsumerConfig(String kafkaConsumerConfig) {
+    return Arrays.stream(kafkaConsumerConfig.split(";"))
+        .map(s -> s.split("="))
+        .map(kv -> Pair.of(kv[0], kv[1]))
+        .collect(Collectors.toMap(Pair::getKey, Pair::getValue));

Review comment:
       nit: If I were writing this I would probably combine these two lines and avoid creating the `Pair`:
   ```suggestion
           .collect(Collectors.toMap(kv -> kv[0], kv -> kv[1]));
   ```
   If you prefer it with the Pair that's fine too.
   
   One thing I think we _should_ change is make this a private static method in `KafkaToPubsub`, since it's only used there.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (3e166d9) into [master](https://codecov.io/gh/apache/beam/commit/0f6af9b41de9bc8b47ec872160f2f547b10b66e9?el=desc) (0f6af9b) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.74%   82.74%   -0.01%     
   ==========================================
     Files         466      466              
     Lines       57525    57527       +2     
   ==========================================
   + Hits        47601    47602       +1     
   - Misses       9924     9925       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `88.09% <0.00%> (-2.39%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.46% <0.00%> (-0.42%)` | :arrow_down: |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
   | [...on/apache\_beam/portability/api/metrics\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.87% <0.00%> (+0.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.81% <0.00%> (+0.26%)` | :arrow_up: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [24179c3...3e166d9](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-752713510


   # [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=h1) Report
   > Merging [#13636](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=desc) (0020886) into [master](https://codecov.io/gh/apache/beam/commit/6c9da02d553cc0d39d29b0293c9b37ac0ceec73e?el=desc) (6c9da02) will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/13636/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #13636      +/-   ##
   ==========================================
   - Coverage   82.76%   82.74%   -0.03%     
   ==========================================
     Files         466      466              
     Lines       57520    57520              
   ==========================================
   - Hits        47605    47593      -12     
   - Misses       9915     9927      +12     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.63% <0.00%> (-4.55%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.90% <0.00%> (-1.82%)` | :arrow_down: |
   | [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.29% <0.00%> (-0.53%)` | :arrow_down: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.57% <0.00%> (-0.42%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13636/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.83% <0.00%> (+0.12%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=footer). Last update [6c9da02...0020886](https://codecov.io/gh/apache/beam/pull/13636?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on pull request #13636:
URL: https://github.com/apache/beam/pull/13636#issuecomment-760941043

Hi @aaltay we tried to turn this test into IT but ran into some troubles, would be great if you could help us with them:
1. When running the test all Docker containers are starting in the machine that is running the test,
so in order for this test to work properly dataflow job should be able to reach test-runner machine by a public IP.
I certainly can't do it on my local machine, not sure how it will behave when running in CI environment.
2. When we pass our fake PubSub url into the dataflow job we receive following error, not sure how this can be avoided, looks like the job will only accept the real Cloud PubSub url:
```json
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "(f214233f9dbe6968): The workflow could not be created. Causes: (f214233f9dbe6719): http://localhost:49169 is not a valid Pub/Sub URL.",
"reason" : "badRequest"
} ],
"message" : "(f214233f9dbe6968): The workflow could not be created. Causes: (f214233f9dbe6719): http://localhost:49169 is not a valid Pub/Sub URL.",
"status" : "INVALID_ARGUMENT"
}
```
FYI @TheNeuralBit

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ramazan-yapparov commented on a change in pull request #13636: [BEAM-11411] [BEAM-11410] Kafka to pub sub E2E test

Posted by GitBox <gi...@apache.org>.

ramazan-yapparov commented on a change in pull request #13636:
URL: https://github.com/apache/beam/pull/13636#discussion_r556481215



##########
File path: examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubE2ETest.java
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.complete.kafkatopubsub;
+
+import static org.apache.beam.examples.complete.kafkatopubsub.transforms.FormatTransform.readFromKafka;
+
+import com.google.auth.Credentials;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Supplier;
+import org.apache.beam.examples.complete.kafkatopubsub.utils.RunKafkaContainer;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Values;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.testcontainers.containers.PubSubEmulatorContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/** E2E test for {@link KafkaToPubsub} pipeline. */
+public class KafkaToPubsubE2ETest {
+
+  @Rule public final transient TestPipeline pipeline = TestPipeline.fromOptions(OPTIONS);
+  @Rule public transient TestPubsubSignal signal = TestPubsubSignal.fromOptions(OPTIONS);
+
+  private static final String PUBSUB_EMULATOR_IMAGE =
+      "gcr.io/google.com/cloudsdktool/cloud-sdk:316.0.0-emulators";
+  private static final String PUBSUB_MESSAGE = "test pubsub message";
+  private static final String PROJECT_ID = "try-kafka-pubsub";
+  private static final String TOPIC_NAME = "listen-to-kafka";
+  private static final PubsubClient.TopicPath TOPIC_PATH =
+      PubsubClient.topicPathFromName(PROJECT_ID, TOPIC_NAME);
+  private static final PipelineOptions OPTIONS = TestPipeline.testingPipelineOptions();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    Credentials credentials = NoopCredentialFactory.fromOptions(OPTIONS).getCredential();
+    OPTIONS.as(GcpOptions.class).setGcpCredential(credentials);
+    OPTIONS.as(GcpOptions.class).setProject(PROJECT_ID);
+    setupPubsubContainer(OPTIONS.as(PubsubOptions.class));
+    createPubsubTopicForTest(OPTIONS.as(PubsubOptions.class));
+  }
+
+  @Test
+  public void testKafkaToPubsubE2E() throws IOException {
+    pipeline.getOptions().as(DirectOptions.class).setBlockOnRun(false);
+
+    RunKafkaContainer rkc = new RunKafkaContainer(PUBSUB_MESSAGE);
+    String bootstrapServer = rkc.getBootstrapServer();
+    String[] kafkaTopicsList = new String[] {rkc.getTopicName()};
+
+    String pubsubTopicPath = TOPIC_PATH.getPath();
+
+    Map<String, Object> kafkaConfig = new HashMap<>();
+    Map<String, String> sslConfig = new HashMap<>();
+
+    PCollection<KV<String, String>> readStrings =
+        pipeline.apply(
+            "readFromKafka",
+            readFromKafka(bootstrapServer, Arrays.asList(kafkaTopicsList), kafkaConfig, sslConfig));
+
+    PCollection<String> readFromPubsub =
+        readStrings
+            .apply(Values.create())
+            .apply("writeToPubSub", PubsubIO.writeStrings().to(pubsubTopicPath))
+            .getPipeline()
+            .apply("readFromPubsub", PubsubIO.readStrings().fromTopic(pubsubTopicPath));
+
+    readFromPubsub.apply(
+        "waitForTestMessage",
+        signal.signalSuccessWhen(
+            readFromPubsub.getCoder(),
+            input -> {
+              if (input == null) {
+                return false;
+              }
+              return input.stream().anyMatch(message -> Objects.equals(message, PUBSUB_MESSAGE));
+            }));
+
+    Supplier<Void> start = signal.waitForStart(Duration.standardSeconds(10));
+    pipeline.apply(signal.signalStart());
+    PipelineResult job = pipeline.run();
+    start.get();
+    signal.waitForSuccess(Duration.standardMinutes(2));
+    try {
+      job.cancel();
+    } catch (IOException | UnsupportedOperationException e) {
+      throw new AssertionError("Could not stop pipeline.", e);
+    }
+  }
+
+  private static void setupPubsubContainer(PubsubOptions options) {
+    PubSubEmulatorContainer emulator =
+        new PubSubEmulatorContainer(DockerImageName.parse(PUBSUB_EMULATOR_IMAGE));
+    emulator.start();
+    String pubsubUrl = emulator.getEmulatorEndpoint();
+    options.setPubsubRootUrl("http://" + pubsubUrl);
+  }
+
+  private static void createPubsubTopicForTest(PubsubOptions options) {
+    try {
+      PubsubClient pubsubClient = PubsubJsonClient.FACTORY.newClient(null, null, options);
+      pubsubClient.createTopic(TOPIC_PATH);
+    } catch (Exception e) {
+      throw new RuntimeException(e);

Review comment:
       Thank you for the suggestion, changed to using `TestPubsub`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org