You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/03/09 16:46:13 UTC

[GitHub] [beam] abergmeier opened a new pull request #14174: [BEAM-XXX] Port join extensions to Python

abergmeier opened a new pull request #14174:
URL: https://github.com/apache/beam/pull/14174


   This is an attempt to port join extensions (Java SDK) to Python.
   
   ---------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam
 .apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.a
 pache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam
 .apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website | Whitespace | Typescript
   --- | --- | --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/be
 am_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794157903


   This is a first naive code attempt. For the following tasks I could really use some guidance with the following questions:
   1. How to handle Tags.
      In Java, the code uses internal knowledge to create what seems to be a class identity? There seem to be some Tags being used in Python but it doesn't seem to have the same power.
      Would be great if someone could shed some light why Tags are there in the first place and how the strategy in Python is.
   
   2. How to handle KV
      I have a hard time finding examples in code that use KV. Since there is a typehint I used this as far as possible in Python code. For the actual KV I then used tuples. Is this fine?
   
   3. Coders
      There seems to be some Coder support but it seems like there is no equivalent to `PCollection.setCoder`. How is this supposed to work in Python?
   
   4. CoGbk*
   There seem to be no further util classes around `CoGbk`*. I assume for the most part will have to implement these.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794165743


   # [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=h1) Report
   > Merging [#14174](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=desc) (b859a99) into [master](https://codecov.io/gh/apache/beam/commit/c391aba4f5edddcd20b25f89e1b987fc482ef129?el=desc) (c391aba) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/14174/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #14174      +/-   ##
   ==========================================
   - Coverage   83.11%   83.10%   -0.01%     
   ==========================================
     Files         469      469              
     Lines       58884    58884              
   ==========================================
   - Hits        48939    48937       -2     
   - Misses       9945     9947       +2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../python/apache\_beam/runners/worker/statesampler.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc3RhdGVzYW1wbGVyLnB5) | | |
   | [...38/build/srcs/sdks/python/apache\_beam/io/avroio.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXZyb2lvLnB5) | | |
   | [...build/srcs/sdks/python/apache\_beam/testing/util.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy91dGlsLnB5) | | |
   | [...srcs/sdks/python/apache\_beam/typehints/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL19faW5pdF9fLnB5) | | |
   | [...ers/dataflow/internal/clients/dataflow/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9jbGllbnRzL2RhdGFmbG93L19faW5pdF9fLnB5) | | |
   | [...ld/srcs/sdks/python/apache\_beam/utils/timestamp.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdGltZXN0YW1wLnB5) | | |
   | [.../srcs/sdks/python/apache\_beam/io/range\_trackers.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vcmFuZ2VfdHJhY2tlcnMucHk=) | | |
   | [...am/testing/benchmarks/nexmark/models/field\_name.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbW9kZWxzL2ZpZWxkX25hbWUucHk=) | | |
   | [...s/sdks/python/apache\_beam/transforms/ptransform.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wdHJhbnNmb3JtLnB5) | | |
   | [.../srcs/sdks/python/apache\_beam/io/azure/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvX19pbml0X18ucHk=) | | |
   | ... and [928 more](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=footer). Last update [c391aba...a968e5c](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794165743


   # [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=h1) Report
   > Merging [#14174](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=desc) (b859a99) into [master](https://codecov.io/gh/apache/beam/commit/c391aba4f5edddcd20b25f89e1b987fc482ef129?el=desc) (c391aba) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/14174/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #14174      +/-   ##
   ==========================================
   - Coverage   83.11%   83.10%   -0.01%     
   ==========================================
     Files         469      469              
     Lines       58884    58884              
   ==========================================
   - Hits        48939    48937       -2     
   - Misses       9945     9947       +2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...tes/tox/py38/build/srcs/sdks/python/test\_config.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vdGVzdF9jb25maWcucHk=) | | |
   | [...ld/srcs/sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | | |
   | [.../apache\_beam/runners/dataflow/internal/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9fX2luaXRfXy5weQ==) | | |
   | [...che\_beam/portability/api/beam\_provision\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcHJvdmlzaW9uX2FwaV9wYjIucHk=) | | |
   | [...38/build/srcs/sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | | |
   | [...apache\_beam/examples/complete/juliaset/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvX19pbml0X18ucHk=) | | |
   | [.../sdks/python/apache\_beam/runners/internal/names.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcm5hbC9uYW1lcy5weQ==) | | |
   | [...y38/build/srcs/sdks/python/apache\_beam/io/vcfio.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vdmNmaW8ucHk=) | | |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | | |
   | [...e\_beam/runners/portability/abstract\_job\_service.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9hYnN0cmFjdF9qb2Jfc2VydmljZS5weQ==) | | |
   | ... and [928 more](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=footer). Last update [c391aba...b859a99](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

codecov[bot] commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794165743


   # [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=h1) Report
   > Merging [#14174](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=desc) (b859a99) into [master](https://codecov.io/gh/apache/beam/commit/c391aba4f5edddcd20b25f89e1b987fc482ef129?el=desc) (c391aba) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/14174/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #14174      +/-   ##
   ==========================================
   - Coverage   83.11%   83.10%   -0.01%     
   ==========================================
     Files         469      469              
     Lines       58884    58884              
   ==========================================
   - Hits        48939    48937       -2     
   - Misses       9945     9947       +2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...d/srcs/sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | | |
   | [.../runners/interactive/testing/test\_cache\_manager.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS90ZXN0aW5nL3Rlc3RfY2FjaGVfbWFuYWdlci5weQ==) | | |
   | [...he\_beam/examples/cookbook/multiple\_output\_pardo.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svbXVsdGlwbGVfb3V0cHV0X3BhcmRvLnB5) | | |
   | [...cs/sdks/python/apache\_beam/portability/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvX19pbml0X18ucHk=) | | |
   | [...y38/build/srcs/sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | | |
   | [...s/python/apache\_beam/examples/cookbook/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svX19pbml0X18ucHk=) | | |
   | [...ild/srcs/sdks/python/apache\_beam/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9tZXRyaWMucHk=) | | |
   | [.../python/apache\_beam/typehints/trivial\_inference.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3RyaXZpYWxfaW5mZXJlbmNlLnB5) | | |
   | [...ild/srcs/sdks/python/apache\_beam/io/gcp/spanner.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3NwYW5uZXIucHk=) | | |
   | [...ython/apache\_beam/runners/direct/bundle\_factory.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvYnVuZGxlX2ZhY3RvcnkucHk=) | | |
   | ... and [928 more](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=footer). Last update [c391aba...b859a99](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794440439


   > [ ] Please create an account at issues.apache.org/jira so we can file a jira and assign it to you
   
   Done. Same username.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aaltay commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

aaltay commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-807509388


   > > We do have a slack channel
   > 
   > I was under the impression that only "official" devs can join that. But perhaps I missed something...
   
   Anyone can join. See slack information on this page: https://beam.apache.org/contribute/
   
   (If this is not clear, we can clarify the documents.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier closed pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier closed pull request #14174:
URL: https://github.com/apache/beam/pull/14174


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794472072


   > > [ ] Please create an account at issues.apache.org/jira so we can file a jira and assign it to you
   > Done. Same username.
   
   @aaltay could you add @abergmeier as a contributor in jira?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794165743


   # [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=h1) Report
   > Merging [#14174](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=desc) (b859a99) into [master](https://codecov.io/gh/apache/beam/commit/c391aba4f5edddcd20b25f89e1b987fc482ef129?el=desc) (c391aba) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/14174/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #14174      +/-   ##
   ==========================================
   - Coverage   83.11%   83.10%   -0.01%     
   ==========================================
     Files         469      469              
     Lines       58884    58884              
   ==========================================
   - Hits        48939    48937       -2     
   - Misses       9945     9947       +2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...python/apache\_beam/runners/portability/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9fX2luaXRfXy5weQ==) | | |
   | [...examples/snippets/transforms/elementwise/values.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS92YWx1ZXMucHk=) | | |
   | [...ache\_beam/runners/interactive/pipeline\_fragment.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9mcmFnbWVudC5weQ==) | | |
   | [.../apache\_beam/io/gcp/datastore/v1new/datastoreio.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9kYXRhc3RvcmVpby5weQ==) | | |
   | [...e\_beam/io/gcp/internal/clients/storage/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2ludGVybmFsL2NsaWVudHMvc3RvcmFnZS9fX2luaXRfXy5weQ==) | | |
   | [...pache\_beam/runners/interactive/caching/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL19faW5pdF9fLnB5) | | |
   | [...snippets/transforms/aggregation/combineglobally.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5lZ2xvYmFsbHkucHk=) | | |
   | [...che\_beam/examples/flink/flink\_streaming\_impulse.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvZmxpbmsvZmxpbmtfc3RyZWFtaW5nX2ltcHVsc2UucHk=) | | |
   | [.../python/apache\_beam/typehints/trivial\_inference.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3RyaXZpYWxfaW5mZXJlbmNlLnB5) | | |
   | [...ild/srcs/sdks/python/apache\_beam/io/filesystems.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZXN5c3RlbXMucHk=) | | |
   | ... and [928 more](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=footer). Last update [c391aba...b859a99](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794165743


   # [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=h1) Report
   > Merging [#14174](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=desc) (b859a99) into [master](https://codecov.io/gh/apache/beam/commit/c391aba4f5edddcd20b25f89e1b987fc482ef129?el=desc) (c391aba) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/14174/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #14174      +/-   ##
   ==========================================
   - Coverage   83.11%   83.10%   -0.01%     
   ==========================================
     Files         469      469              
     Lines       58884    58884              
   ==========================================
   - Hits        48939    48937       -2     
   - Misses       9945     9947       +2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ache\_beam/portability/api/beam\_job\_api\_pb2\_grpc.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fam9iX2FwaV9wYjJfZ3JwYy5weQ==) | | |
   | [.../srcs/sdks/python/apache\_beam/dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3NjaGVtYXMucHk=) | | |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | | |
   | [...beam/testing/benchmarks/nexmark/queries/query11.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTExLnB5) | | |
   | [...hon/apache\_beam/portability/api/schema\_pb2\_grpc.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL3NjaGVtYV9wYjJfZ3JwYy5weQ==) | | |
   | [.../examples/snippets/transforms/aggregation/count.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb3VudC5weQ==) | | |
   | [...nippets/transforms/aggregation/groupintobatches.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9ncm91cGludG9iYXRjaGVzLnB5) | | |
   | [...sdks/python/apache\_beam/portability/common\_urns.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvY29tbW9uX3VybnMucHk=) | | |
   | [...rcs/sdks/python/apache\_beam/typehints/typecheck.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3R5cGVjaGVjay5weQ==) | | |
   | [...srcs/sdks/python/apache\_beam/io/gcp/dicomclient.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RpY29tY2xpZW50LnB5) | | |
   | ... and [928 more](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=footer). Last update [c391aba...f1360c4](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier commented on a change in pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier commented on a change in pull request #14174:
URL: https://github.com/apache/beam/pull/14174#discussion_r590697011



##########
File path: sdks/python/apache_beam/transforms/join/keyedpcollectiontuple.py
##########
@@ -0,0 +1,162 @@
+
+from typing import Any, Generic, List, TypeVar
+import apache_beam as beam
+from apache_beam.typehints import typehints
+from apache_beam.coders.coders import Coder
+
+K = TypeVar("K")
+V = TypeVar("V")
+InputT = TypeVar("InputT")
+OutputT = TypeVar("OutputT")
+
+class KeyedPCollectionTuple(Generic[K]):

Review comment:
       Thanks. Just figured that out and already pushed some updates.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794482495


   > In Python we always infer the coder from the PCollection's element_type, which is determined from typehints. Often we just fall back to FastPrimitivesCoder.
   
   TBH infering/relying on (implicit) typehints sounds scary to me. Will give this a try and report back.
   
   > In your case it should be enough to just use typehints.KV.
   
   I hope they are correct now. For now I tried fully qualifying all typehints.
   
   > As discussed elsewhere, we generally just use python primitives rather than the CoGbk* util classes. Let me know if there are specific ones that don't seem to be supported in Python.
   
   Yes thanks, indeed for now it seems I got rid of all of these.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-869860344


   So our code diverged from something general so much that I won't make sense for general use case.
   Not even sure this was a good idea to begin with - at least for GCP.
   Thus will close this now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-869867339


   > So our code diverged from something general so much that I won't make sense for general use case.
   > Not even sure this was a good idea to begin with - at least for GCP.
   > Thus will close this now.
   
   Ack, thank you for following up on this. If you feel anything you've built would make sense to generalize a contribution is always welcome :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-869860344


   So our code diverged from something general so much that I won't make sense for general use case.
   Not even sure this was a good idea to begin with - at least for GCP.
   Thus will close this now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier closed pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier closed pull request #14174:
URL: https://github.com/apache/beam/pull/14174


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794165743


   # [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=h1) Report
   > Merging [#14174](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=desc) (b859a99) into [master](https://codecov.io/gh/apache/beam/commit/c391aba4f5edddcd20b25f89e1b987fc482ef129?el=desc) (c391aba) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/14174/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #14174      +/-   ##
   ==========================================
   - Coverage   83.11%   83.10%   -0.01%     
   ==========================================
     Files         469      469              
     Lines       58884    58884              
   ==========================================
   - Hits        48939    48937       -2     
   - Misses       9945     9947       +2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ow/dataflow\_exercise\_streaming\_metrics\_pipeline.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19leGVyY2lzZV9zdHJlYW1pbmdfbWV0cmljc19waXBlbGluZS5weQ==) | | |
   | [.../srcs/sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=) | | |
   | [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | | |
   | [...\_beam/testing/benchmarks/nexmark/queries/query7.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTcucHk=) | | |
   | [...rcs/sdks/python/apache\_beam/runners/direct/util.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdXRpbC5weQ==) | | |
   | [...d/srcs/sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | | |
   | [...n/apache\_beam/runners/direct/evaluation\_context.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXZhbHVhdGlvbl9jb250ZXh0LnB5) | | |
   | [...ython/apache\_beam/typehints/decorators\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL2RlY29yYXRvcnNfdGVzdF9weTMucHk=) | | |
   | [...thon/apache\_beam/runners/portability/job\_server.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9qb2Jfc2VydmVyLnB5) | | |
   | [...amples/snippets/transforms/aggregation/distinct.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9kaXN0aW5jdC5weQ==) | | |
   | ... and [928 more](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=footer). Last update [c391aba...9217d84](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794165743


   # [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=h1) Report
   > Merging [#14174](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=desc) (b859a99) into [master](https://codecov.io/gh/apache/beam/commit/c391aba4f5edddcd20b25f89e1b987fc482ef129?el=desc) (c391aba) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/14174/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #14174      +/-   ##
   ==========================================
   - Coverage   83.11%   83.10%   -0.01%     
   ==========================================
     Files         469      469              
     Lines       58884    58884              
   ==========================================
   - Hits        48939    48937       -2     
   - Misses       9945     9947       +2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...cs/sdks/python/apache\_beam/typehints/decorators.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL2RlY29yYXRvcnMucHk=) | | |
   | [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | | |
   | [.../srcs/sdks/python/apache\_beam/dataframe/convert.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2NvbnZlcnQucHk=) | | |
   | [...apache\_beam/examples/cookbook/custom\_ptransform.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY3VzdG9tX3B0cmFuc2Zvcm0ucHk=) | | |
   | [...xamples/snippets/transforms/elementwise/flatmap.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9mbGF0bWFwLnB5) | | |
   | [...ache\_beam/coders/proto2\_coder\_test\_messages\_pb2.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Byb3RvMl9jb2Rlcl90ZXN0X21lc3NhZ2VzX3BiMi5weQ==) | | |
   | [...s/python/apache\_beam/typehints/sharded\_key\_type.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NoYXJkZWRfa2V5X3R5cGUucHk=) | | |
   | [...cs/sdks/python/apache\_beam/io/gcp/gcsfilesystem.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2djc2ZpbGVzeXN0ZW0ucHk=) | | |
   | [...amples/snippets/transforms/aggregation/distinct.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9kaXN0aW5jdC5weQ==) | | |
   | [...ks/python/apache\_beam/io/gcp/datastore/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS9fX2luaXRfXy5weQ==) | | |
   | ... and [928 more](https://codecov.io/gh/apache/beam/pull/14174/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=footer). Last update [c391aba...b859a99](https://codecov.io/gh/apache/beam/pull/14174?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on a change in pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on a change in pull request #14174:
URL: https://github.com/apache/beam/pull/14174#discussion_r590679919



##########
File path: sdks/python/apache_beam/transforms/join/keyedpcollectiontuple.py
##########
@@ -0,0 +1,162 @@
+
+from typing import Any, Generic, List, TypeVar
+import apache_beam as beam
+from apache_beam.typehints import typehints
+from apache_beam.coders.coders import Coder
+
+K = TypeVar("K")
+V = TypeVar("V")
+InputT = TypeVar("InputT")
+OutputT = TypeVar("OutputT")
+
+class KeyedPCollectionTuple(Generic[K]):

Review comment:
       I think it should be possible to add the Join library to Python without porting `KeyedPCollectionTuple`. In Java this is necessary for CoGroupByKey, but in Python we represent tagged/keyed PCollections as `Dict[str, PCollection]`, you can see how this works with CoGroupByKey in that transform's API docs: https://beam.apache.org/releases/pydoc/2.28.0/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794471320


   Sorry I completely missed the questions you asked!
   
   > 1. How to handle Tags.
   ~~In Java, the code uses internal knowledge to create what seems to be a class identity? There seem to be some Tags being used in Python but it doesn't seem to have the same power.
   Would be great if someone could shed some light why Tags are there in the first place and how the strategy in Python is.~~
   Since this produces a simpler dict in Python, will hardcode the strings.
   
   :+1: 
   
   > 2. How to handle KV
   I have a hard time finding examples in code that use KV. Since there is a typehint I used this as far as possible in Python code. For the actual KV I then used tuples. Is this fine?
   
   Yes 2-tuples are the preferred way to represent KVs at execution time in Python.
   
   > 3. Coders
   There seems to be some Coder support but it seems like there is no equivalent to PCollection.setCoder. How is this supposed to work in Python?
   
   In Python we always infer the coder from the PCollection's element_type, which is determined from typehints. Often we just fall back to FastPrimitivesCoder. In your case it should be enough to just use `typehints.KV`.
   
   > 4. CoGbk*
   There seem to be no further util classes around CoGbk*. I assume for the most part will have to implement these.
   
   As discussed elsewhere, we generally just use python primitives rather than the CoGbk* util classes. Let me know if there are specific ones that don't seem to be supported in Python.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-797334912


   Could anyone point to where
   
   > (left, right)
         | "Do me" >> LeftOuterJoin(nullValue=None)
   
   could be enabled? I saw that this is done somehow for:
   
   > Dict | str >> CoGroupByKey()
   
   But I am not sure how the `|` gets enabled by code.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aaltay commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

aaltay commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-807305091


   > > What would be the next step on this PR?
   > 
   > We decided to evolve the code in our internal app first (thus no public code changes for now).
   > 
   > We tested out various API designs and landed on one that we internally find pleasing.
   > 
   > Now we started writing tests to ensure various aspects are working and coverage is there.
   > In parallel we ensured that the code is already quite well type annotated.
   > 
   > Especially with type annotation we now found several aspects to be lacking in decorator code.
   > I am currently trying to understand what magic beam is trying to apply and why some of our cases don't work out.
   > 
   > The thing that currently is most annoying to me is that there is no public chat to have a quick talk about things like design decision in Beam. The whole development process seems a bit heavy handed :(
   
   I understand.
   
   We do have a slack channel, but not all relevant people are online there all the time. You should be able to schedule a time with relevant folks to have a quick chat on slack though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-799763220


   Sorry for not getting back sooner. PTransform has implementations for the operators: :https://github.com/apache/beam/blob/2306bd0733711f8c7807d059256f9c5b0432f5f7/sdks/python/apache_beam/transforms/ptransform.py#L539
   
   It just passes "pvalueish" directly to the PTransform's `expand` implementation. I don't think it pulls apart tuples or dicts, they're just passed through directly, so you'll need to handle them as CoGroupByKey does: https://github.com/apache/beam/blob/2306bd0733711f8c7807d059256f9c5b0432f5f7/sdks/python/apache_beam/transforms/util.py#L190-L202
   
   It would be worth generalizing this somehow.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-807369079


   > We do have a slack channel
   
   I was under the impression that only "official" devs can join that. But perhaps I missed something...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aaltay commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

aaltay commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-797713304


   > > > [ ] Please create an account at issues.apache.org/jira so we can file a jira and assign it to you
   > > > Done. Same username.
   > 
   > @aaltay could you add @abergmeier as a contributor in jira?
   
   Done. Sorry for the delay.
   
   Added both abergmeier and abergmeieratotto. Looks like both are your accounts.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier edited a comment on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier edited a comment on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794157903

This is a first naive code attempt. For the following tasks I could really use some guidance with the following questions:
1. How to handle Tags.
~~In Java, the code uses internal knowledge to create what seems to be a class identity? There seem to be some Tags being used in Python but it doesn't seem to have the same power.
Would be great if someone could shed some light why Tags are there in the first place and how the strategy in Python is.~~
Since this produces a simpler `dict` in Python, will hardcode the strings.

2. How to handle KV
I have a hard time finding examples in code that use KV. Since there is a typehint I used this as far as possible in Python code. For the actual KV I then used tuples. Is this fine?

3. Coders
There seems to be some Coder support but it seems like there is no equivalent to `PCollection.setCoder`. How is this supposed to work in Python?

4. CoGbk*
There seem to be no further util classes around `CoGbk`*. I assume for the most part will have to implement these.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-807227759


   > What would be the next step on this PR?
   
   We decided to evolve the code in our internal app first (thus no public code changes for now).
   
   We tested out various API designs and landed on one that we internally find pleasing.
   
   Now we started writing tests to ensure various aspects are working and coverage is there.
   In parallel we ensured that the code is already quite well type annotated.
   
   Especially with type annotation we now found several aspects to be lacking in decorator code.
   I am currently trying to understand what magic beam is trying to apply and why some of our cases don't work out.
   
   The thing that currently is most annoying to me is that there is no public chat to have a quick talk about things like design decision in Beam. The whole development process seems a bit heavy handed :(


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aaltay commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

aaltay commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-807154269


   @abergmeier - What would be the next step on this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794146728


   R: @aaltay
   R: @charlesccychen


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

TheNeuralBit commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-869867339


   > So our code diverged from something general so much that I won't make sense for general use case.
   > Not even sure this was a good idea to begin with - at least for GCP.
   > Thus will close this now.
   
   Ack, thank you for following up on this. If you feel anything you've built would make sense to generalize a contribution is always welcome :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] abergmeier commented on pull request #14174: [BEAM-XXX] Port join extensions to Python

Posted by GitBox <gi...@apache.org>.

abergmeier commented on pull request #14174:
URL: https://github.com/apache/beam/pull/14174#issuecomment-794447458


   > [ ] I think it would make sense to put everything for this effort in apache_beam.transforms.join, since we don't split out extensions for the Python SDK.
   
   Done.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org