You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/09 19:18:54 UTC
[GitHub] [beam] rohdesamuel opened a new pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
rohdesamuel opened a new pull request #12799:
URL: https://github.com/apache/beam/pull/12799
This adds record_pipeline and clear to the RecordingManager.
record_pipeline() allows to start a recording without going through show() or collect()
clear() clears the cache of all recordings.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `62.13%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.31% -0.04%
==========================================
Files 450 451 +1
Lines 53708 53878 +170
==========================================
+ Hits 44230 44349 +119
- Misses 9478 9529 +51
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <85.71%> (ø)` | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <100.00%> (ø)` | |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <100.00%> (ø)` | |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| ... and [32 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [5206131...0ebd2c2](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696302427
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.05%`.
> The diff coverage is `62.50%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.30% -0.06%
==========================================
Files 450 451 +1
Lines 53708 53849 +141
==========================================
+ Hits 44230 44319 +89
- Misses 9478 9530 +52
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <85.71%> (ø)` | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <100.00%> (ø)` | |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <100.00%> (ø)` | |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| ... and [29 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [5206131...c2686e1](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #12799:
URL: https://github.com/apache/beam/pull/12799
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `72.34%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.31% -0.05%
==========================================
Files 450 451 +1
Lines 53708 53878 +170
==========================================
+ Hits 44230 44347 +117
- Misses 9478 9531 +53
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/snowflake.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc25vd2ZsYWtlLnB5) | `64.15% <ø> (ø)` | |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.53% <ø> (-0.21%)` | :arrow_down: |
| [...apache\_beam/runners/portability/portable\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9ydW5uZXIucHk=) | `77.37% <ø> (ø)` | |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `87.39% <20.00%> (-3.03%)` | :arrow_down: |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.16% <66.66%> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `89.90% <83.33%> (-0.59%)` | :arrow_down: |
| ... and [38 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [126554d...f119319](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696205611
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `62.50%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.31% -0.04%
==========================================
Files 450 451 +1
Lines 53708 53880 +172
==========================================
+ Hits 44230 44351 +121
- Misses 9478 9529 +51
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <85.71%> (ø)` | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <100.00%> (ø)` | |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <100.00%> (ø)` | |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [5206131...c2686e1](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `72.34%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.31% -0.05%
==========================================
Files 450 451 +1
Lines 53708 53878 +170
==========================================
+ Hits 44230 44347 +117
- Misses 9478 9531 +53
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/snowflake.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc25vd2ZsYWtlLnB5) | `64.15% <ø> (ø)` | |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.53% <ø> (-0.21%)` | :arrow_down: |
| [...apache\_beam/runners/portability/portable\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9ydW5uZXIucHk=) | `77.37% <ø> (ø)` | |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `87.39% <20.00%> (-3.03%)` | :arrow_down: |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.16% <66.66%> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `89.90% <83.33%> (-0.59%)` | :arrow_down: |
| ... and [38 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [126554d...f119319](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/a2126e571cfddc90e3819e66ef1f2da3d52833ed?el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `58.56%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.36% 82.30% -0.07%
==========================================
Files 450 451 +1
Lines 53708 53849 +141
==========================================
+ Hits 44238 44319 +81
- Misses 9470 9530 +60
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.33% <100.00%> (+5.47%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.68% <0.00%> (-1.23%)` | :arrow_down: |
| ... and [19 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [5206131...9ba76c5](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `72.34%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.31% -0.05%
==========================================
Files 450 451 +1
Lines 53708 53878 +170
==========================================
+ Hits 44230 44347 +117
- Misses 9478 9531 +53
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/snowflake.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc25vd2ZsYWtlLnB5) | `64.15% <ø> (ø)` | |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.53% <ø> (-0.21%)` | :arrow_down: |
| [...apache\_beam/runners/portability/portable\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9ydW5uZXIucHk=) | `77.37% <ø> (ø)` | |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `87.39% <20.00%> (-3.03%)` | :arrow_down: |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.16% <66.66%> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `89.90% <83.33%> (-0.59%)` | :arrow_down: |
| ... and [38 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [126554d...f119319](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696426921
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #12799:
URL: https://github.com/apache/beam/pull/12799#discussion_r489494335
##########
File path: sdks/python/apache_beam/runners/interactive/utils.py
##########
@@ -34,7 +34,8 @@ def to_element_list(
reader, # type: Generator[Union[TestStreamPayload.Event, WindowedValueHolder]]
coder, # type: Coder
include_window_info, # type: bool
- n=None # type: int
+ n=None, # type: int
+ include_teststream_events=False, # type: bool
Review comment:
In this case, 'teststream events' seem t refer to watermark/processing time events only. Perhaps you can call it 'include_time_events' or something like that? (since Data events also come form the teststream but are ont affected by this flag)
##########
File path: sdks/python/apache_beam/runners/interactive/recording_manager.py
##########
@@ -256,7 +259,7 @@ def describe(self):
size = sum(
cache_manager.size('full', s.cache_key) for s in self._streams.values())
- return {'size': size, 'start': self._start}
+ return {'size': size}
Review comment:
Why are you removing start from here?
##########
File path: sdks/python/apache_beam/runners/interactive/recording_manager.py
##########
@@ -314,15 +330,55 @@ def cancel(self):
r.wait_until_finish()
self._recordings = set()
+ # The recordings rely on a reference to the BCJ to correctly finish. So we
+ # evict the BCJ after they complete.
+ ie.current_env().evict_background_caching_job(self.user_pipeline)
+
def describe(self):
# type: () -> dict[str, int]
"""Returns a dictionary describing the cache and recording."""
+ cache_manager = ie.current_env().get_cache_manager(self.user_pipeline)
+ capture_size = getattr(cache_manager, 'capture_size', 0)
+
descriptions = [r.describe() for r in self._recordings]
- size = sum(d['size'] for d in descriptions)
- start = min(d['start'] for d in descriptions)
- return {'size': size, 'start': start}
+ if descriptions:
+ size = sum(d['size'] for d in descriptions) + capture_size
+ else:
+ size = capture_size
Review comment:
You can just add line 347 without the if/else. sum(..) over an empty list will return 0.
##########
File path: sdks/python/apache_beam/runners/interactive/recording_manager.py
##########
@@ -114,14 +113,19 @@ def read(self, tail=True):
# all elements from the cache were read. In the latter situation, it may be
# the case that the pipeline was still running. Thus, another invocation of
# `read` will yield new elements.
+ count_limiter = CountLimiter(self._n)
+ time_limiter = ProcessingTimeLimiter(self._duration_secs)
+ limiters = (count_limiter, time_limiter)
for e in utils.to_element_list(reader,
coder,
include_window_info=True,
- n=self._n):
- for l in limiters:
- l.update(e)
-
- yield e
+ n=self._n,
+ include_teststream_events=True):
+ if isinstance(e, TestStreamPayload.Event):
+ time_limiter.update(e)
+ else:
+ count_limiter.update(e)
+ yield e
Review comment:
Are Data elements sent decoded, and that's why we have this if/else?
##########
File path: sdks/python/apache_beam/runners/interactive/recording_manager_test.py
##########
@@ -149,43 +152,37 @@ def test_read_n(self):
def test_read_duration(self):
"""Test that the stream only reads a 'duration' of elements."""
+ def as_windowed_value(element):
+ return WindowedValueHolder(WindowedValue(element, 0, []))
values = (FileRecordsBuilder(tag=self.cache_key)
.advance_processing_time(1)
- .add_element(element=0, event_time_secs=0)
+ .add_element(element=as_windowed_value(0), event_time_secs=0)
.advance_processing_time(1)
- .add_element(element=1, event_time_secs=1)
+ .add_element(element=as_windowed_value(1), event_time_secs=1)
.advance_processing_time(1)
- .add_element(element=2, event_time_secs=3)
+ .add_element(element=as_windowed_value(2), event_time_secs=3)
.advance_processing_time(1)
- .add_element(element=3, event_time_secs=4)
+ .add_element(element=as_windowed_value(3), event_time_secs=4)
.advance_processing_time(1)
- .add_element(element=4, event_time_secs=5)
+ .add_element(element=as_windowed_value(4), event_time_secs=5)
.build()) # yapf: disable
+ values = [
+ v.recorded_event for v in values if isinstance(v, TestStreamFileRecord)
+ ]
+
self.mock_result.set_state(PipelineState.DONE)
self.cache.write(values, 'full', self.cache_key)
- self.cache.save_pcoder(None, 'full', self.cache_key)
-
- # The elements read from the cache are TestStreamFileRecord instances and
- # have the underlying elements encoded. This method decodes the elements
- # from the TestStreamFileRecord.
- def get_elements(events):
- coder = coders.FastPrimitivesCoder()
- elements = []
- for e in events:
- if not isinstance(e, TestStreamFileRecord):
- continue
-
- if e.recorded_event.element_event:
- elements += ([
- coder.decode(el.encoded_element)
- for el in e.recorded_event.element_event.elements
- ])
- return elements
+ self.cache.save_pcoder(coders.FastPrimitivesCoder(), 'full', self.cache_key)
# The following tests a progression of reading different durations from the
# cache.
+
+ # Small convienence function for getting the values.
+ def get_elements(events):
+ return [e.value for e in events]
Review comment:
I am trying to figure out - where did we stop passing encoded data events?
##########
File path: sdks/python/apache_beam/runners/interactive/recording_manager.py
##########
@@ -314,15 +330,55 @@ def cancel(self):
r.wait_until_finish()
self._recordings = set()
+ # The recordings rely on a reference to the BCJ to correctly finish. So we
+ # evict the BCJ after they complete.
+ ie.current_env().evict_background_caching_job(self.user_pipeline)
+
def describe(self):
# type: () -> dict[str, int]
"""Returns a dictionary describing the cache and recording."""
+ cache_manager = ie.current_env().get_cache_manager(self.user_pipeline)
+ capture_size = getattr(cache_manager, 'capture_size', 0)
+
descriptions = [r.describe() for r in self._recordings]
- size = sum(d['size'] for d in descriptions)
- start = min(d['start'] for d in descriptions)
- return {'size': size, 'start': start}
+ if descriptions:
+ size = sum(d['size'] for d in descriptions) + capture_size
+ else:
+ size = capture_size
+ start = self._start_time_sec
+ bcj = ie.current_env().get_background_caching_job(self.user_pipeline)
+ if bcj:
+ state = bcj.state
+ else:
+ state = PipelineState.STOPPED
+ return {'size': size, 'start': start, 'state': state}
+
+ def record_pipeline(self):
+ # type: () -> bool
+
+ """Starts a background caching job for this RecordingManager's pipeline."""
+
+ runner = self.user_pipeline.runner
+ if isinstance(runner, ir.InteractiveRunner):
+ runner = runner._underlying_runner
+
+ # Make sure that sources without a user reference are still cached.
+ pi.watch_sources(self.user_pipeline)
+
+ # Attempt to run background caching job to record any sources.
+ if ie.current_env().is_in_ipython:
+ warnings.filterwarnings(
+ 'ignore',
+ 'options is deprecated since First stable release. References to '
+ '<pipeline>.options will not be supported',
+ category=DeprecationWarning)
+ if bcj.attempt_to_run_background_caching_job(
+ runner, self.user_pipeline, options=self.user_pipeline.options):
+ self._start_time_sec = time.time()
+ return True
+ return False
Review comment:
Would this failure be logged / handled somewhere? Maybe we should log the fialure to start the BCJ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696870959
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696328692
ah it looks like it's broken on master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/4d7df122ca053828bd2c7dffa9f390ba2bd8678e?el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
=======================================
Coverage 82.30% 82.31%
=======================================
Files 451 451
Lines 53849 53878 +29
=======================================
+ Hits 44318 44347 +29
Misses 9531 9531
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.16% <0.00%> (+0.17%)` | :arrow_up: |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `100.00% <0.00%> (+8.00%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [126554d...f119319](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696905097
thanks Sam!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on a change in pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #12799:
URL: https://github.com/apache/beam/pull/12799#discussion_r489790682
##########
File path: sdks/python/apache_beam/runners/interactive/recording_manager.py
##########
@@ -314,15 +330,55 @@ def cancel(self):
r.wait_until_finish()
self._recordings = set()
+ # The recordings rely on a reference to the BCJ to correctly finish. So we
+ # evict the BCJ after they complete.
+ ie.current_env().evict_background_caching_job(self.user_pipeline)
+
def describe(self):
# type: () -> dict[str, int]
"""Returns a dictionary describing the cache and recording."""
+ cache_manager = ie.current_env().get_cache_manager(self.user_pipeline)
+ capture_size = getattr(cache_manager, 'capture_size', 0)
+
descriptions = [r.describe() for r in self._recordings]
- size = sum(d['size'] for d in descriptions)
- start = min(d['start'] for d in descriptions)
- return {'size': size, 'start': start}
+ if descriptions:
+ size = sum(d['size'] for d in descriptions) + capture_size
+ else:
+ size = capture_size
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `72.34%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.31% -0.05%
==========================================
Files 450 451 +1
Lines 53708 53878 +170
==========================================
+ Hits 44230 44347 +117
- Misses 9478 9531 +53
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/snowflake.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc25vd2ZsYWtlLnB5) | `64.15% <ø> (ø)` | |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.53% <ø> (-0.21%)` | :arrow_down: |
| [...apache\_beam/runners/portability/portable\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9ydW5uZXIucHk=) | `77.37% <ø> (ø)` | |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `87.39% <20.00%> (-3.03%)` | :arrow_down: |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.16% <66.66%> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `89.90% <83.33%> (-0.59%)` | :arrow_down: |
| ... and [38 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [126554d...f119319](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/4d7df122ca053828bd2c7dffa9f390ba2bd8678e?el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
=======================================
Coverage 82.30% 82.31%
=======================================
Files 451 451
Lines 53849 53878 +29
=======================================
+ Hits 44318 44347 +29
Misses 9531 9531
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.16% <0.00%> (+0.17%)` | :arrow_up: |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `100.00% <0.00%> (+8.00%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [126554d...f119319](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696905097
thanks Sam!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.05%`.
> The diff coverage is `62.13%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.30% -0.06%
==========================================
Files 450 451 +1
Lines 53708 53911 +203
==========================================
+ Hits 44230 44369 +139
- Misses 9478 9542 +64
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <85.71%> (ø)` | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <100.00%> (ø)` | |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <100.00%> (ø)` | |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| ... and [33 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [5206131...0ebd2c2](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-693725469
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696269197
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696247296
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696205611
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **increase** coverage by `0.37%`.
> The diff coverage is `75.13%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
+ Coverage 82.35% 82.72% +0.37%
==========================================
Files 450 454 +4
Lines 53708 55787 +2079
==========================================
+ Hits 44230 46149 +1919
- Misses 9478 9638 +160
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/snowflake.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc25vd2ZsYWtlLnB5) | `64.15% <ø> (ø)` | |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.74% <ø> (ø)` | |
| [...apache\_beam/runners/portability/portable\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9ydW5uZXIucHk=) | `77.37% <ø> (ø)` | |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `87.39% <20.00%> (-3.03%)` | :arrow_down: |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.16% <66.66%> (+0.01%)` | :arrow_up: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <85.71%> (ø)` | |
| ... and [50 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [126554d...f119319](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/a2126e571cfddc90e3819e66ef1f2da3d52833ed?el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `58.56%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.36% 82.30% -0.07%
==========================================
Files 450 451 +1
Lines 53708 53849 +141
==========================================
+ Hits 44238 44319 +81
- Misses 9470 9530 +60
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.33% <100.00%> (+5.47%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.68% <0.00%> (-1.23%)` | :arrow_down: |
| ... and [19 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [5206131...9ba76c5](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692976118
R: @pabloem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on a change in pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #12799:
URL: https://github.com/apache/beam/pull/12799#discussion_r489792184
##########
File path: sdks/python/apache_beam/runners/interactive/recording_manager.py
##########
@@ -314,15 +330,55 @@ def cancel(self):
r.wait_until_finish()
self._recordings = set()
+ # The recordings rely on a reference to the BCJ to correctly finish. So we
+ # evict the BCJ after they complete.
+ ie.current_env().evict_background_caching_job(self.user_pipeline)
+
def describe(self):
# type: () -> dict[str, int]
"""Returns a dictionary describing the cache and recording."""
+ cache_manager = ie.current_env().get_cache_manager(self.user_pipeline)
+ capture_size = getattr(cache_manager, 'capture_size', 0)
+
descriptions = [r.describe() for r in self._recordings]
- size = sum(d['size'] for d in descriptions)
- start = min(d['start'] for d in descriptions)
- return {'size': size, 'start': start}
+ if descriptions:
+ size = sum(d['size'] for d in descriptions) + capture_size
+ else:
+ size = capture_size
+ start = self._start_time_sec
+ bcj = ie.current_env().get_background_caching_job(self.user_pipeline)
+ if bcj:
+ state = bcj.state
+ else:
+ state = PipelineState.STOPPED
+ return {'size': size, 'start': start, 'state': state}
+
+ def record_pipeline(self):
+ # type: () -> bool
+
+ """Starts a background caching job for this RecordingManager's pipeline."""
+
+ runner = self.user_pipeline.runner
+ if isinstance(runner, ir.InteractiveRunner):
+ runner = runner._underlying_runner
+
+ # Make sure that sources without a user reference are still cached.
+ pi.watch_sources(self.user_pipeline)
+
+ # Attempt to run background caching job to record any sources.
+ if ie.current_env().is_in_ipython:
+ warnings.filterwarnings(
+ 'ignore',
+ 'options is deprecated since First stable release. References to '
+ '<pipeline>.options will not be supported',
+ category=DeprecationWarning)
+ if bcj.attempt_to_run_background_caching_job(
+ runner, self.user_pipeline, options=self.user_pipeline.options):
+ self._start_time_sec = time.time()
+ return True
+ return False
Review comment:
I don't think we should because it's not necessarily an error that the BCJ didn't start. There's logic inside the BCJ that only starts if it needs to, i.e. the list unbounded sources changed or there is nothing in the cache. In the next PR, there is extra more logging around when it is OK to try and start a new recording.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on a change in pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #12799:
URL: https://github.com/apache/beam/pull/12799#discussion_r489793686
##########
File path: sdks/python/apache_beam/runners/interactive/recording_manager_test.py
##########
@@ -149,43 +152,37 @@ def test_read_n(self):
def test_read_duration(self):
"""Test that the stream only reads a 'duration' of elements."""
+ def as_windowed_value(element):
+ return WindowedValueHolder(WindowedValue(element, 0, []))
values = (FileRecordsBuilder(tag=self.cache_key)
.advance_processing_time(1)
- .add_element(element=0, event_time_secs=0)
+ .add_element(element=as_windowed_value(0), event_time_secs=0)
.advance_processing_time(1)
- .add_element(element=1, event_time_secs=1)
+ .add_element(element=as_windowed_value(1), event_time_secs=1)
.advance_processing_time(1)
- .add_element(element=2, event_time_secs=3)
+ .add_element(element=as_windowed_value(2), event_time_secs=3)
.advance_processing_time(1)
- .add_element(element=3, event_time_secs=4)
+ .add_element(element=as_windowed_value(3), event_time_secs=4)
.advance_processing_time(1)
- .add_element(element=4, event_time_secs=5)
+ .add_element(element=as_windowed_value(4), event_time_secs=5)
.build()) # yapf: disable
+ values = [
+ v.recorded_event for v in values if isinstance(v, TestStreamFileRecord)
+ ]
+
self.mock_result.set_state(PipelineState.DONE)
self.cache.write(values, 'full', self.cache_key)
- self.cache.save_pcoder(None, 'full', self.cache_key)
-
- # The elements read from the cache are TestStreamFileRecord instances and
- # have the underlying elements encoded. This method decodes the elements
- # from the TestStreamFileRecord.
- def get_elements(events):
- coder = coders.FastPrimitivesCoder()
- elements = []
- for e in events:
- if not isinstance(e, TestStreamFileRecord):
- continue
-
- if e.recorded_event.element_event:
- elements += ([
- coder.decode(el.encoded_element)
- for el in e.recorded_event.element_event.elements
- ])
- return elements
+ self.cache.save_pcoder(coders.FastPrimitivesCoder(), 'full', self.cache_key)
# The following tests a progression of reading different durations from the
# cache.
+
+ # Small convienence function for getting the values.
+ def get_elements(events):
+ return [e.value for e in events]
Review comment:
The test uses the `InMemoryCache` set in the `setUp` method that directly writes values to an in-memory map verbatim.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696426921
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/4d7df122ca053828bd2c7dffa9f390ba2bd8678e?el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.30% 82.30% -0.01%
==========================================
Files 451 451
Lines 53849 53911 +62
==========================================
+ Hits 44318 44369 +51
- Misses 9531 9542 +11
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.31% <100.00%> (+0.26%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/snowflake.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc25vd2ZsYWtlLnB5) | `60.46% <0.00%> (-3.69%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.74% <0.00%> (+0.20%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.52% <0.00%> (+0.53%)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [126554d...f119319](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696206366
ok thanks Sam. LGTM!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `72.34%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.31% -0.05%
==========================================
Files 450 451 +1
Lines 53708 53878 +170
==========================================
+ Hits 44230 44347 +117
- Misses 9478 9531 +53
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/snowflake.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc25vd2ZsYWtlLnB5) | `64.15% <ø> (ø)` | |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.53% <ø> (-0.21%)` | :arrow_down: |
| [...apache\_beam/runners/portability/portable\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9ydW5uZXIucHk=) | `77.37% <ø> (ø)` | |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `87.39% <20.00%> (-3.03%)` | :arrow_down: |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.16% <66.66%> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `89.90% <83.33%> (-0.59%)` | :arrow_down: |
| ... and [38 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [126554d...f119319](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on a change in pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #12799:
URL: https://github.com/apache/beam/pull/12799#discussion_r489784307
##########
File path: sdks/python/apache_beam/runners/interactive/utils.py
##########
@@ -34,7 +34,8 @@ def to_element_list(
reader, # type: Generator[Union[TestStreamPayload.Event, WindowedValueHolder]]
coder, # type: Coder
include_window_info, # type: bool
- n=None # type: int
+ n=None, # type: int
+ include_teststream_events=False, # type: bool
Review comment:
Gotcha, changed to include_time_events
##########
File path: sdks/python/apache_beam/runners/interactive/recording_manager.py
##########
@@ -114,14 +113,19 @@ def read(self, tail=True):
# all elements from the cache were read. In the latter situation, it may be
# the case that the pipeline was still running. Thus, another invocation of
# `read` will yield new elements.
+ count_limiter = CountLimiter(self._n)
+ time_limiter = ProcessingTimeLimiter(self._duration_secs)
+ limiters = (count_limiter, time_limiter)
for e in utils.to_element_list(reader,
coder,
include_window_info=True,
- n=self._n):
- for l in limiters:
- l.update(e)
-
- yield e
+ n=self._n,
+ include_teststream_events=True):
+ if isinstance(e, TestStreamPayload.Event):
+ time_limiter.update(e)
+ else:
+ count_limiter.update(e)
+ yield e
Review comment:
Yep, it's to make sure we only count decoded elements. I added a comment to make it more clear.
##########
File path: sdks/python/apache_beam/runners/interactive/recording_manager.py
##########
@@ -256,7 +259,7 @@ def describe(self):
size = sum(
cache_manager.size('full', s.cache_key) for s in self._streams.values())
- return {'size': size, 'start': self._start}
+ return {'size': size}
Review comment:
Because the start time wasn't correct if we only start a background caching job. In that case there wouldn't be a new `Recording` so the start time would be 0. I think this also cleans up the logic a bit (no more `min`-ing over all the start times).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-694388520
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #12799:
URL: https://github.com/apache/beam/pull/12799
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-693711277
> The failing test seems to be a related test in Windows: https://github.com/apache/beam/pull/12799/checks?check_run_id=1119719710
> Can you check whether it's related to this PR (and fix if so), and if not still try to send in a fix in a new PR?
Fixed it! The InteractiveEnvironment was being recreated for every test which works if everything is in serial. But the tests are run in parallel so tests were destroying each other's environments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-693435551
The failing test seems to be a related test in Windows: https://github.com/apache/beam/pull/12799/checks?check_run_id=1119719710
Can you check whether it's related to this PR (and fix if so), and if not still try to send in a fix in a new PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-694484142
> > The failing test seems to be a related test in Windows: https://github.com/apache/beam/pull/12799/checks?check_run_id=1119719710
> > Can you check whether it's related to this PR (and fix if so), and if not still try to send in a fix in a new PR?
>
> Fixed it! The InteractiveEnvironment was being recreated for every test which works if everything is in serial. But the tests are run in parallel so tests were destroying each other's environments.
Brian pinged me on the flaky tests so I moved it out of this PR and into https://github.com/apache/beam/pull/12866
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `62.13%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.31% -0.04%
==========================================
Files 450 451 +1
Lines 53708 53878 +170
==========================================
+ Hits 44230 44349 +119
- Misses 9478 9529 +51
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <85.71%> (ø)` | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <100.00%> (ø)` | |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <100.00%> (ø)` | |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| ... and [32 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [5206131...0ebd2c2](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696247296
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-696206475
I'll merge after precommit passes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `72.34%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.31% -0.05%
==========================================
Files 450 451 +1
Lines 53708 53878 +170
==========================================
+ Hits 44230 44347 +117
- Misses 9478 9531 +53
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/snowflake.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc25vd2ZsYWtlLnB5) | `64.15% <ø> (ø)` | |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.53% <ø> (-0.21%)` | :arrow_down: |
| [...apache\_beam/runners/portability/portable\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9wb3J0YWJsZV9ydW5uZXIucHk=) | `77.37% <ø> (ø)` | |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `87.39% <20.00%> (-3.03%)` | :arrow_down: |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.16% <66.66%> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `89.90% <83.33%> (-0.59%)` | :arrow_down: |
| ... and [38 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [126554d...f119319](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/a2126e571cfddc90e3819e66ef1f2da3d52833ed?el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `58.56%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.36% 82.30% -0.07%
==========================================
Files 450 451 +1
Lines 53708 53849 +141
==========================================
+ Hits 44238 44319 +81
- Misses 9470 9530 +60
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.33% <100.00%> (+5.47%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.38% <100.00%> (+0.30%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `88.68% <0.00%> (-1.23%)` | :arrow_down: |
| ... and [19 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [5206131...9ba76c5](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `62.13%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.31% -0.04%
==========================================
Files 450 451 +1
Lines 53708 53878 +170
==========================================
+ Hits 44230 44349 +119
- Misses 9478 9529 +51
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <85.71%> (ø)` | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <100.00%> (ø)` | |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <100.00%> (ø)` | |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| ... and [32 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [5206131...0ebd2c2](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.05%`.
> The diff coverage is `62.13%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.30% -0.06%
==========================================
Files 450 451 +1
Lines 53708 53911 +203
==========================================
+ Hits 44230 44369 +139
- Misses 9478 9542 +64
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <85.71%> (ø)` | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <100.00%> (ø)` | |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <100.00%> (ø)` | |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| ... and [33 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [5206131...0ebd2c2](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12799: [BEAM-10603] Add record_pipeline, clear to RM and fix duration limiter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12799:
URL: https://github.com/apache/beam/pull/12799#issuecomment-692960218
# [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=h1) Report
> Merging [#12799](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `62.50%`.
[](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12799 +/- ##
==========================================
- Coverage 82.35% 82.31% -0.04%
==========================================
Files 450 451 +1
Lines 53708 53880 +172
==========================================
+ Hits 44230 44351 +121
- Misses 9478 9529 +51
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.52% <ø> (+1.73%)` | :arrow_up: |
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <85.71%> (ø)` | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <100.00%> (ø)` | |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <100.00%> (ø)` | |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `90.47% <100.00%> (-3.08%)` | :arrow_down: |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/12799/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=footer). Last update [5206131...0ebd2c2](https://codecov.io/gh/apache/beam/pull/12799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org