You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/01/29 23:21:25 UTC
[GitHub] [beam] yifanmai opened a new pull request #13851: [BEAM-11644] Add ValidatesRunner test for translations.pack_combiners
yifanmai opened a new pull request #13851:
URL: https://github.com/apache/beam/pull/13851
Add an integration test using TestPipeline with packable combiners. Combiner packing will be triggered if and only if the TestPipeline runner enables it.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam
.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #13851:
URL: https://github.com/apache/beam/pull/13851#discussion_r567172005
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/translations_test.py
##########
@@ -227,6 +232,53 @@ def assert_is_topologically_sorted(transform_id, visited_pcolls):
assert_is_topologically_sorted(
optimized_pipeline_proto.root_transform_ids[0], set())
+ @attr('ValidatesRunner')
+ def test_run_packable_combine_per_key(self):
+ class MultipleCombines(beam.PTransform):
+ def expand(self, pcoll):
+ assert_that(
Review comment:
For example, Dataflow will not trigger it unless experiment "pre_optimize" is set, right ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-770109030
# [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=h1) Report
> Merging [#13851](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=desc) (97cc14a) into [master](https://codecov.io/gh/apache/beam/commit/73731ec4f3f2d185e89aa3e378d321c2154ecf53?el=desc) (73731ec) will **increase** coverage by `0.08%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13851 +/- ##
==========================================
+ Coverage 82.84% 82.92% +0.08%
==========================================
Files 466 466
Lines 57596 58268 +672
==========================================
+ Hits 47713 48321 +608
- Misses 9883 9947 +64
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.55% <0.00%> (-0.27%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.79% <0.00%> (-0.21%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.38% <0.00%> (+0.13%)` | :arrow_up: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.77% <0.00%> (+0.26%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.73% <0.00%> (+0.45%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `91.31% <0.00%> (+1.79%)` | :arrow_up: |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `96.29% <0.00%> (+2.44%)` | :arrow_up: |
| ... and [3 more](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=footer). Last update [73731ec...97cc14a](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-771865293
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-770109030
# [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=h1) Report
> Merging [#13851](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=desc) (97cc14a) into [master](https://codecov.io/gh/apache/beam/commit/73731ec4f3f2d185e89aa3e378d321c2154ecf53?el=desc) (73731ec) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13851 +/- ##
==========================================
- Coverage 82.84% 82.83% -0.01%
==========================================
Files 466 466
Lines 57596 57596
==========================================
- Hits 47713 47712 -1
- Misses 9883 9884 +1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.22% <0.00%> (-0.64%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.55% <0.00%> (-0.27%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `91.31% <0.00%> (+1.79%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=footer). Last update [73731ec...97cc14a](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13851: [BEAM-11644] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-770109030
# [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=h1) Report
> Merging [#13851](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=desc) (a9b999a) into [master](https://codecov.io/gh/apache/beam/commit/73731ec4f3f2d185e89aa3e378d321c2154ecf53?el=desc) (73731ec) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13851 +/- ##
==========================================
- Coverage 82.84% 82.83% -0.02%
==========================================
Files 466 466
Lines 57596 57596
==========================================
- Hits 47713 47707 -6
- Misses 9883 9889 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.56% <0.00%> (-0.42%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.55% <0.00%> (-0.27%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.57% <0.00%> (-0.26%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.35% <0.00%> (+0.35%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=footer). Last update [73731ec...a9b999a](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #13851:
URL: https://github.com/apache/beam/pull/13851#discussion_r567171701
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/translations_test.py
##########
@@ -227,6 +232,53 @@ def assert_is_topologically_sorted(transform_id, visited_pcolls):
assert_is_topologically_sorted(
optimized_pipeline_proto.root_transform_ids[0], set())
+ @attr('ValidatesRunner')
+ def test_run_packable_combine_per_key(self):
+ class MultipleCombines(beam.PTransform):
+ def expand(self, pcoll):
+ assert_that(
Review comment:
How do we confirm that combiner packing was triggered here ? This would pass even without combiner packing, right ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj merged pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
chamikaramj merged pull request #13851:
URL: https://github.com/apache/beam/pull/13851
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13851: [BEAM-11644] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-770109030
# [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=h1) Report
> Merging [#13851](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=desc) (a9b999a) into [master](https://codecov.io/gh/apache/beam/commit/73731ec4f3f2d185e89aa3e378d321c2154ecf53?el=desc) (73731ec) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13851 +/- ##
==========================================
- Coverage 82.84% 82.83% -0.02%
==========================================
Files 466 466
Lines 57596 57596
==========================================
- Hits 47713 47707 -6
- Misses 9883 9889 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.56% <0.00%> (-0.42%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.55% <0.00%> (-0.27%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.57% <0.00%> (-0.26%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.35% <0.00%> (+0.35%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=footer). Last update [73731ec...a9b999a](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-771865293
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-770109030
# [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=h1) Report
> Merging [#13851](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=desc) (97cc14a) into [master](https://codecov.io/gh/apache/beam/commit/73731ec4f3f2d185e89aa3e378d321c2154ecf53?el=desc) (73731ec) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13851 +/- ##
==========================================
- Coverage 82.84% 82.83% -0.01%
==========================================
Files 466 466
Lines 57596 57596
==========================================
- Hits 47713 47712 -1
- Misses 9883 9884 +1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.22% <0.00%> (-0.64%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.55% <0.00%> (-0.27%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `91.31% <0.00%> (+1.79%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=footer). Last update [73731ec...97cc14a](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-770109030
# [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=h1) Report
> Merging [#13851](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=desc) (97cc14a) into [master](https://codecov.io/gh/apache/beam/commit/73731ec4f3f2d185e89aa3e378d321c2154ecf53?el=desc) (73731ec) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13851 +/- ##
==========================================
- Coverage 82.84% 82.83% -0.01%
==========================================
Files 466 466
Lines 57596 57596
==========================================
- Hits 47713 47712 -1
- Misses 9883 9884 +1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `86.45% <0.00%> (-1.05%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.22% <0.00%> (-0.64%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.55% <0.00%> (-0.27%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `91.31% <0.00%> (+1.79%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=footer). Last update [73731ec...97cc14a](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-770109030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #13851: [BEAM-11644] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-770109030
# [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=h1) Report
> Merging [#13851](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=desc) (a9b999a) into [master](https://codecov.io/gh/apache/beam/commit/73731ec4f3f2d185e89aa3e378d321c2154ecf53?el=desc) (73731ec) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13851 +/- ##
==========================================
- Coverage 82.84% 82.83% -0.02%
==========================================
Files 466 466
Lines 57596 57596
==========================================
- Hits 47713 47707 -6
- Misses 9883 9889 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.56% <0.00%> (-0.42%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.55% <0.00%> (-0.27%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.57% <0.00%> (-0.26%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.35% <0.00%> (+0.35%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=footer). Last update [73731ec...a9b999a](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-770109030
# [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=h1) Report
> Merging [#13851](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=desc) (a9b999a) into [master](https://codecov.io/gh/apache/beam/commit/73731ec4f3f2d185e89aa3e378d321c2154ecf53?el=desc) (73731ec) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13851 +/- ##
==========================================
- Coverage 82.84% 82.83% -0.02%
==========================================
Files 466 466
Lines 57596 57596
==========================================
- Hits 47713 47707 -6
- Misses 9883 9889 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `92.85% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `91.56% <0.00%> (-0.42%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.55% <0.00%> (-0.27%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.57% <0.00%> (-0.26%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `80.35% <0.00%> (+0.35%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=footer). Last update [73731ec...97cc14a](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj merged pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
chamikaramj merged pull request #13851:
URL: https://github.com/apache/beam/pull/13851
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yifanmai commented on a change in pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
yifanmai commented on a change in pull request #13851:
URL: https://github.com/apache/beam/pull/13851#discussion_r567175544
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/translations_test.py
##########
@@ -227,6 +232,53 @@ def assert_is_topologically_sorted(transform_id, visited_pcolls):
assert_is_topologically_sorted(
optimized_pipeline_proto.root_transform_ids[0], set())
+ @attr('ValidatesRunner')
+ def test_run_packable_combine_per_key(self):
+ class MultipleCombines(beam.PTransform):
+ def expand(self, pcoll):
+ assert_that(
Review comment:
Yes this would test without combiner packing. This test serves a different purpose; it tests that packable combines can be run successfully with correct outputs, whether or not the packing actually happens. I updated the comment to clarify this.
The test that combiner packing was triggered has to be runner specific because not every runner supports combiner packing yet:
- DataflowRunner has a test [here](https://github.com/apache/beam/blob/a44768f074ad0c55d5ca19c8ec6a47887b7e8538/sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py#L852-L883) but only checks for it under an experiment flag.
- FnApiRunner has a test [here] but it is disabled because combiner packing doesn't actually work for PortableRunner and its subclasses yet.
Speaking of experiment flags, ValidatesRunner has the downside of only testing the case with no experiment flags. So it won't do anything useful until combiner packing is enabled by default by each runner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-770109030
# [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=h1) Report
> Merging [#13851](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=desc) (97cc14a) into [master](https://codecov.io/gh/apache/beam/commit/73731ec4f3f2d185e89aa3e378d321c2154ecf53?el=desc) (73731ec) will **increase** coverage by `0.08%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #13851 +/- ##
==========================================
+ Coverage 82.84% 82.92% +0.08%
==========================================
Files 466 466
Lines 57596 58268 +672
==========================================
+ Hits 47713 48321 +608
- Misses 9883 9947 +64
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `84.55% <0.00%> (-0.27%)` | :arrow_down: |
| [...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==) | `79.79% <0.00%> (-0.21%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.70% <0.00%> (-0.13%)` | :arrow_down: |
| [setup.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.38% <0.00%> (+0.13%)` | :arrow_up: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.77% <0.00%> (+0.26%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.73% <0.00%> (+0.45%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.82% <0.00%> (+0.52%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `91.31% <0.00%> (+1.79%)` | :arrow_up: |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `96.29% <0.00%> (+2.44%)` | :arrow_up: |
| ... and [3 more](https://codecov.io/gh/apache/beam/pull/13851/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=footer). Last update [73731ec...97cc14a](https://codecov.io/gh/apache/beam/pull/13851?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yifanmai commented on pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
yifanmai commented on pull request #13851:
URL: https://github.com/apache/beam/pull/13851#issuecomment-770132169
cc @robertwb
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yifanmai commented on a change in pull request #13851: [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners
Posted by GitBox <gi...@apache.org>.
yifanmai commented on a change in pull request #13851:
URL: https://github.com/apache/beam/pull/13851#discussion_r567175544
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner/translations_test.py
##########
@@ -227,6 +232,53 @@ def assert_is_topologically_sorted(transform_id, visited_pcolls):
assert_is_topologically_sorted(
optimized_pipeline_proto.root_transform_ids[0], set())
+ @attr('ValidatesRunner')
+ def test_run_packable_combine_per_key(self):
+ class MultipleCombines(beam.PTransform):
+ def expand(self, pcoll):
+ assert_that(
Review comment:
Yes this would test without combiner packing. This test serves a different purpose; it tests that packable combines can be run successfully with correct outputs, whether or not the packing actually happens. I updated the comment to clarify this.
The test that combiner packing was triggered has to be runner specific because not every runner supports combiner packing yet:
- DataflowRunner has a test [here](https://github.com/apache/beam/blob/a44768f074ad0c55d5ca19c8ec6a47887b7e8538/sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py#L852-L883) but only checks for it under an experiment flag.
- FnApiRunner has a test [here](https://github.com/apache/beam/blob/a44768f074ad0c55d5ca19c8ec6a47887b7e8538/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner_test.py#L1037-L1040) but it is disabled because combiner packing doesn't actually work for PortableRunner and its subclasses yet.
Speaking of experiment flags, ValidatesRunner has the downside of only testing the case with no experiment flags. So it won't do anything useful until combiner packing is enabled by default by each runner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org