You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/27 19:01:48 UTC
[GitHub] [beam] rohdesamuel opened a new pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
rohdesamuel opened a new pull request #12704:
URL: https://github.com/apache/beam/pull/12704
Change-Id: I660b98ac7cd562eb03aab6d15d829eae8726ebad
The following API changes the way that the Interactive Runner caches
data from unbounded sources. Previously, a user-configurable duration
would run a "Background Caching Job" that would cache all elements from
unbounded source. Then, the user pipeline would run by reading from
that cache after completion. Now, there is a long lived "Background
Recording Job". This replaces the caching job with a background pipeline
that records elements from the unbounded sources and most importantly
does not block user pipeline execution.
The following APIs have changed:
show(*pcolls, include_window_info=False, visualize_data=False) -->
show(*pcolls, n='inf', duration='inf', include_window_info=False, visualize_data=False)
The show() command was modified to include a max number of elements and
a max duration of elements to read. The show() command will end when
either of the limiters gets triggered, whichever comes first. If a
limiter is not supplied it is assumed to be infinite. For example,
show(pcoll, n=5000) will read and block until 5000 elements are read.
collect(pcoll, include_window_info=False) -->
collect(pcoll, n='inf', duration='inf', include_window_info=False)
The collect() command was modified to include a max number of elements and
a max duration of elements to read. The collect() command will end when
either of the limiters gets triggered, whichever comes first. If a
limiter is not supplied it is assumed to be infinite. For example,
show(pcoll, n=5000) will read and block until 5000 elements are read.
head() was removed.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace
--- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689048887
# [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=h1) Report
> Merging [#12704](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2bb60c323095340240ec4982c1e9dabc397107e5?el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `16.80%`.
[](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
==========================================
- Coverage 40.43% 40.40% -0.04%
==========================================
Files 449 451 +2
Lines 53530 53685 +155
==========================================
+ Hits 21643 21689 +46
- Misses 31887 31996 +109
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `28.57% <0.00%> (-0.19%)` | :arrow_down: |
| [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `20.20% <0.00%> (ø)` | |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `27.92% <0.00%> (-1.14%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `28.06% <0.00%> (+0.04%)` | :arrow_up: |
| [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `34.61% <6.97%> (+5.37%)` | :arrow_up: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `24.34% <9.09%> (-0.66%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `24.21% <12.50%> (-0.55%)` | :arrow_down: |
| ... and [5 more](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=footer). Last update [1463ff0...d78dee0](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-690533642
R: @pabloem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689048887
# [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=h1) Report
> Merging [#12704](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/9f4d54a2e60ec45150437c0b050f4c73cca91f36?el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `11.62%`.
[](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
==========================================
+ Coverage 40.23% 40.28% +0.04%
==========================================
Files 455 455
Lines 53731 53792 +61
==========================================
+ Hits 21619 21670 +51
- Misses 32112 32122 +10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `40.32% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `34.61% <6.97%> (+5.37%)` | :arrow_up: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `24.21% <12.50%> (-0.55%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `28.08% <15.62%> (-1.52%)` | :arrow_down: |
| [...he\_beam/runners/interactive/pipeline\_instrument.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9pbnN0cnVtZW50LnB5) | `14.86% <50.00%> (+0.16%)` | :arrow_up: |
| [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `28.92% <0.00%> (+0.24%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `43.73% <0.00%> (+0.76%)` | :arrow_up: |
| [sdks/python/apache\_beam/transforms/core.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb3JlLnB5) | `39.22% <0.00%> (+0.94%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=footer). Last update [9f4d54a...332aa62](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-692746763
LGTM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689048887
# [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=h1) Report
> Merging [#12704](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/9f4d54a2e60ec45150437c0b050f4c73cca91f36?el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `11.62%`.
[](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
==========================================
+ Coverage 40.23% 40.28% +0.04%
==========================================
Files 455 455
Lines 53731 53792 +61
==========================================
+ Hits 21619 21670 +51
- Misses 32112 32122 +10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `40.32% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `34.61% <6.97%> (+5.37%)` | :arrow_up: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `24.21% <12.50%> (-0.55%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `28.08% <15.62%> (-1.52%)` | :arrow_down: |
| [...he\_beam/runners/interactive/pipeline\_instrument.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9pbnN0cnVtZW50LnB5) | `14.86% <50.00%> (+0.16%)` | :arrow_up: |
| [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `28.92% <0.00%> (+0.24%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `43.73% <0.00%> (+0.76%)` | :arrow_up: |
| [sdks/python/apache\_beam/transforms/core.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb3JlLnB5) | `39.22% <0.00%> (+0.94%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=footer). Last update [9f4d54a...332aa62](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689048887
# [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=h1) Report
> Merging [#12704](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2bb60c323095340240ec4982c1e9dabc397107e5?el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `16.80%`.
[](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
==========================================
- Coverage 40.43% 40.40% -0.04%
==========================================
Files 449 451 +2
Lines 53530 53685 +155
==========================================
+ Hits 21643 21689 +46
- Misses 31887 31996 +109
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `28.57% <0.00%> (-0.19%)` | :arrow_down: |
| [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `20.20% <0.00%> (ø)` | |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `27.92% <0.00%> (-1.14%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `28.06% <0.00%> (+0.04%)` | :arrow_up: |
| [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `34.61% <6.97%> (+5.37%)` | :arrow_up: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `24.34% <9.09%> (-0.66%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `24.21% <12.50%> (-0.55%)` | :arrow_down: |
| ... and [5 more](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=footer). Last update [1463ff0...d78dee0](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689044081
R: @davidyan74
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689048887
# [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=h1) Report
> Merging [#12704](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/b45a1b6a10de795c07016bc4d5b1c5cca76b3624?el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `74.75%`.
[](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
=======================================
Coverage 82.38% 82.38%
=======================================
Files 451 451
Lines 53721 53713 -8
=======================================
- Hits 44256 44251 -5
+ Misses 9465 9462 -3
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.05% <84.61%> (+3.45%)` | :arrow_up: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.78% <90.90%> (-1.38%)` | :arrow_down: |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-0.46%)` | :arrow_down: |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.74% <0.00%> (+0.10%)` | :arrow_up: |
| [.../runners/portability/fn\_api\_runner/translations.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3RyYW5zbGF0aW9ucy5weQ==) | `88.17% <0.00%> (+0.11%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.34% <0.00%> (+0.35%)` | :arrow_up: |
| ... and [5 more](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=footer). Last update [1463ff0...71d4a90](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on a change in pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #12704:
URL: https://github.com/apache/beam/pull/12704#discussion_r486566707
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -342,121 +343,79 @@ def show(*pcolls, **configs):
assert isinstance(pcoll, beam.pvalue.PCollection), (
'{} is not an apache_beam.pvalue.PCollection.'.format(pcoll))
user_pipeline = pcolls[0].pipeline
- for pcoll in pcolls:
- assert pcoll.pipeline is user_pipeline, (
- '{} belongs to a different user-defined pipeline ({}) than that of'
- ' other PCollections ({}).'.format(
- pcoll, pcoll.pipeline, user_pipeline))
+
# TODO(BEAM-8288): Remove below pops and assertion once Python 2 is
# deprecated from Beam.
include_window_info = configs.pop('include_window_info', False)
visualize_data = configs.pop('visualize_data', False)
+ n = configs.pop('n', 'inf')
+ duration = configs.pop('duration', 'inf')
+
+ if n == 'inf':
+ n = float('inf')
+
+ if duration == 'inf':
+ duration = float('inf')
Review comment:
Added assertions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12704:
URL: https://github.com/apache/beam/pull/12704#discussion_r490431373
##########
File path: sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
##########
@@ -407,13 +392,7 @@ def _display_dataframe(self, data, update=None):
self._is_datatable_empty = False
def _to_dataframe(self):
- results = []
- cache_manager = ie.current_env().get_cache_manager(self._pcoll.pipeline)
- if cache_manager.exists('full', self._cache_key):
- coder = cache_manager.load_pcoder('full', self._cache_key)
- reader, _ = cache_manager.read('full', self._cache_key)
- results = list(to_element_list(reader, coder, include_window_info=True))
-
Review comment:
This looks like a likely culprit, the error message in the flakes references a `full` directory in the cache:
```
E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full\\ac8879590f-2021876280456-2021876278608-2021914046928'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on a change in pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #12704:
URL: https://github.com/apache/beam/pull/12704#discussion_r486566349
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -273,6 +270,10 @@ def show(*pcolls, **configs):
The given pcolls can be dictionary of PCollections (as values), or iterable
of PCollections or plain PCollection values.
+ The user can specify either the max number of elements with `n` to read
+ or the maximum duration of elements to read with `duration`. When a limiter is
+ not supplied, it is assumed to be infinite.
+
There are 2 boolean configurations:
Review comment:
Done, added some arg doc strings.
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -342,121 +343,79 @@ def show(*pcolls, **configs):
assert isinstance(pcoll, beam.pvalue.PCollection), (
'{} is not an apache_beam.pvalue.PCollection.'.format(pcoll))
user_pipeline = pcolls[0].pipeline
- for pcoll in pcolls:
- assert pcoll.pipeline is user_pipeline, (
- '{} belongs to a different user-defined pipeline ({}) than that of'
- ' other PCollections ({}).'.format(
- pcoll, pcoll.pipeline, user_pipeline))
+
# TODO(BEAM-8288): Remove below pops and assertion once Python 2 is
# deprecated from Beam.
include_window_info = configs.pop('include_window_info', False)
visualize_data = configs.pop('visualize_data', False)
+ n = configs.pop('n', 'inf')
+ duration = configs.pop('duration', 'inf')
+
+ if n == 'inf':
+ n = float('inf')
+
+ if duration == 'inf':
+ duration = float('inf')
+
# This assertion is to protect the backward compatibility for function
# signature change after Python 2 deprecation.
assert not configs, (
'The only configs supported are include_window_info and '
'visualize_data.')
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on a change in pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
KevinGG commented on a change in pull request #12704:
URL: https://github.com/apache/beam/pull/12704#discussion_r486514105
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -342,121 +343,79 @@ def show(*pcolls, **configs):
assert isinstance(pcoll, beam.pvalue.PCollection), (
'{} is not an apache_beam.pvalue.PCollection.'.format(pcoll))
user_pipeline = pcolls[0].pipeline
- for pcoll in pcolls:
- assert pcoll.pipeline is user_pipeline, (
- '{} belongs to a different user-defined pipeline ({}) than that of'
- ' other PCollections ({}).'.format(
- pcoll, pcoll.pipeline, user_pipeline))
+
# TODO(BEAM-8288): Remove below pops and assertion once Python 2 is
# deprecated from Beam.
include_window_info = configs.pop('include_window_info', False)
visualize_data = configs.pop('visualize_data', False)
+ n = configs.pop('n', 'inf')
+ duration = configs.pop('duration', 'inf')
+
+ if n == 'inf':
+ n = float('inf')
+
+ if duration == 'inf':
+ duration = float('inf')
+
# This assertion is to protect the backward compatibility for function
# signature change after Python 2 deprecation.
assert not configs, (
'The only configs supported are include_window_info and '
'visualize_data.')
Review comment:
Include the new configs `n` and `duration`.
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -273,6 +270,10 @@ def show(*pcolls, **configs):
The given pcolls can be dictionary of PCollections (as values), or iterable
of PCollections or plain PCollection values.
+ The user can specify either the max number of elements with `n` to read
+ or the maximum duration of elements to read with `duration`. When a limiter is
+ not supplied, it is assumed to be infinite.
+
There are 2 boolean configurations:
Review comment:
Nit: how about merging these configurations in the docstrings.
Something like:
```
There are 4 configurations:
#. include_window_info=<True/False>. If True, windowing information of the
data will be visualized too. Default is false.
#. visualize_data=<True/False>. By default, the visualization contains data
tables rendering data from given pcolls separately as if they are
converted into dataframes. If visualize_data is True, there will be a
more dive-in widget and statistically overview widget of the data.
Otherwise, those 2 data visualization widgets will not be displayed.
#. n=<int>. Max number of elements to visualize. Default 'inf'.
#. duration=<int>. Max duration of elements to read. Default 'inf'.
```
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -342,121 +343,79 @@ def show(*pcolls, **configs):
assert isinstance(pcoll, beam.pvalue.PCollection), (
'{} is not an apache_beam.pvalue.PCollection.'.format(pcoll))
user_pipeline = pcolls[0].pipeline
- for pcoll in pcolls:
- assert pcoll.pipeline is user_pipeline, (
- '{} belongs to a different user-defined pipeline ({}) than that of'
- ' other PCollections ({}).'.format(
- pcoll, pcoll.pipeline, user_pipeline))
+
# TODO(BEAM-8288): Remove below pops and assertion once Python 2 is
# deprecated from Beam.
include_window_info = configs.pop('include_window_info', False)
visualize_data = configs.pop('visualize_data', False)
+ n = configs.pop('n', 'inf')
+ duration = configs.pop('duration', 'inf')
+
+ if n == 'inf':
+ n = float('inf')
+
+ if duration == 'inf':
+ duration = float('inf')
+
# This assertion is to protect the backward compatibility for function
# signature change after Python 2 deprecation.
assert not configs, (
'The only configs supported are include_window_info and '
'visualize_data.')
- runner = user_pipeline.runner
- if isinstance(runner, ir.InteractiveRunner):
- runner = runner._underlying_runner
-
- # Make sure that sources without a user reference are still cached.
- pi.watch_sources(user_pipeline)
-
- # Make sure that all PCollections to be shown are watched. If a PCollection
- # has not been watched, make up a variable name for that PCollection and watch
- # it. No validation is needed here because the watch logic can handle
- # arbitrary variables.
- watched_pcollections = set()
- for watching in ie.current_env().watching():
- for _, val in watching:
- if hasattr(val, '__class__') and isinstance(val, beam.pvalue.PCollection):
- watched_pcollections.add(val)
- for pcoll in pcolls:
- if pcoll not in watched_pcollections:
- watch({'anonymous_pcollection_{}'.format(id(pcoll)): pcoll})
-
- if ie.current_env().is_in_ipython:
- warnings.filterwarnings(
- 'ignore',
- 'options is deprecated since First stable release. References to '
- '<pipeline>.options will not be supported',
- category=DeprecationWarning)
- # Attempt to run background caching job since we have the reference to the
- # user-defined pipeline.
- bcj.attempt_to_run_background_caching_job(
- runner, user_pipeline, user_pipeline.options)
-
- pcolls = set(pcolls)
- computed_pcolls = set()
- for pcoll in pcolls:
- if pcoll in ie.current_env().computed_pcollections:
- computed_pcolls.add(pcoll)
- pcolls = pcolls.difference(computed_pcolls)
- # If in notebook, static plotting computed pcolls as computation is done.
- if ie.current_env().is_in_notebook:
- for pcoll in computed_pcolls:
- visualize(
- pcoll,
- include_window_info=include_window_info,
- display_facets=visualize_data)
- elif ie.current_env().is_in_ipython:
- for pcoll in computed_pcolls:
- visualize(pcoll, include_window_info=include_window_info)
-
- if not pcolls:
- return
-
- # Build a pipeline fragment for the PCollections and run it.
- result = pf.PipelineFragment(list(pcolls), user_pipeline.options).run()
- ie.current_env().set_pipeline_result(user_pipeline, result)
-
- # If in notebook, dynamic plotting as computation goes.
- if ie.current_env().is_in_notebook:
- for pcoll in pcolls:
- visualize(
- pcoll,
- dynamic_plotting_interval=1,
- include_window_info=include_window_info,
- display_facets=visualize_data)
-
- # Invoke wait_until_finish to ensure the blocking nature of this API without
- # relying on the run to be blocking.
- result.wait_until_finish()
-
- # If just in ipython shell, plotting once when the computation is completed.
- if ie.current_env().is_in_ipython and not ie.current_env().is_in_notebook:
- for pcoll in pcolls:
- visualize(pcoll, include_window_info=include_window_info)
-
- # If the pipeline execution is successful at this stage, mark the computation
- # completeness for the given PCollections so that when further `show`
- # invocation occurs, Interactive Beam wouldn't need to re-compute them.
- if result.state is beam.runners.runner.PipelineState.DONE:
- ie.current_env().mark_pcollection_computed(pcolls)
-
-
-def collect(pcoll, include_window_info=False):
- """Materializes all of the elements from a PCollection into a Dataframe.
- For example::
+ recording_manager = RecordingManager(user_pipeline)
+ recording = recording_manager.record(
+ pcolls, max_n=n, max_duration_secs=duration)
+
+ # Catch a KeyboardInterrupt to gracefully cancel the recording and
+ # visualizations.
+ try:
+ # If in notebook, static plotting computed pcolls as computation is done.
+ if ie.current_env().is_in_notebook:
+ for stream in recording.computed().values():
+ visualize(
+ stream,
+ include_window_info=include_window_info,
+ display_facets=visualize_data)
+ elif ie.current_env().is_in_ipython:
+ for stream in recording.computed().values():
+ visualize(stream, include_window_info=include_window_info)
Review comment:
I assume you've verified that, when the recording is not computed, this (static) and below (with dynamic_plotting_interval) `visualize` statements will update the same visualization. Is it because you are generating a consistent display_id from the recording manager?
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -342,121 +343,79 @@ def show(*pcolls, **configs):
assert isinstance(pcoll, beam.pvalue.PCollection), (
'{} is not an apache_beam.pvalue.PCollection.'.format(pcoll))
user_pipeline = pcolls[0].pipeline
- for pcoll in pcolls:
- assert pcoll.pipeline is user_pipeline, (
- '{} belongs to a different user-defined pipeline ({}) than that of'
- ' other PCollections ({}).'.format(
- pcoll, pcoll.pipeline, user_pipeline))
+
# TODO(BEAM-8288): Remove below pops and assertion once Python 2 is
# deprecated from Beam.
include_window_info = configs.pop('include_window_info', False)
visualize_data = configs.pop('visualize_data', False)
+ n = configs.pop('n', 'inf')
+ duration = configs.pop('duration', 'inf')
+
+ if n == 'inf':
+ n = float('inf')
+
+ if duration == 'inf':
+ duration = float('inf')
Review comment:
Do we need any assertion in the future? Like `n` needs to be a positive integer. And `duration` will be a string such as `1h2m3s`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689048887
# [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=h1) Report
> Merging [#12704](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/4635afbf686b216983d1c0552b33cd435f922812?el=desc) will **decrease** coverage by `0.14%`.
> The diff coverage is `11.62%`.
[](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
==========================================
- Coverage 40.42% 40.28% -0.15%
==========================================
Files 450 455 +5
Lines 53665 53792 +127
==========================================
- Hits 21696 21670 -26
- Misses 31969 32122 +153
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `40.32% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `34.61% <6.97%> (+5.37%)` | :arrow_up: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `24.21% <12.50%> (-0.55%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `28.08% <15.62%> (-0.98%)` | :arrow_down: |
| [...he\_beam/runners/interactive/pipeline\_instrument.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9pbnN0cnVtZW50LnB5) | `14.86% <50.00%> (ø)` | |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `26.77% <0.00%> (-0.34%)` | :arrow_down: |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `24.23% <0.00%> (-0.05%)` | :arrow_down: |
| [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `20.20% <0.00%> (ø)` | |
| ... and [9 more](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=footer). Last update [6213d47...a2301e1](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689048887
# [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=h1) Report
> Merging [#12704](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/9f4d54a2e60ec45150437c0b050f4c73cca91f36?el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `11.62%`.
[](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
==========================================
+ Coverage 40.23% 40.28% +0.04%
==========================================
Files 455 455
Lines 53731 53792 +61
==========================================
+ Hits 21619 21670 +51
- Misses 32112 32122 +10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `40.32% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `34.61% <6.97%> (+5.37%)` | :arrow_up: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `24.21% <12.50%> (-0.55%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `28.08% <15.62%> (-1.52%)` | :arrow_down: |
| [...he\_beam/runners/interactive/pipeline\_instrument.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9pbnN0cnVtZW50LnB5) | `14.86% <50.00%> (+0.16%)` | :arrow_up: |
| [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `28.92% <0.00%> (+0.24%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `43.73% <0.00%> (+0.76%)` | :arrow_up: |
| [sdks/python/apache\_beam/transforms/core.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb3JlLnB5) | `39.22% <0.00%> (+0.94%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=footer). Last update [9f4d54a...332aa62](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on a change in pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
KevinGG commented on a change in pull request #12704:
URL: https://github.com/apache/beam/pull/12704#discussion_r490593309
##########
File path: sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
##########
@@ -407,13 +392,7 @@ def _display_dataframe(self, data, update=None):
self._is_datatable_empty = False
def _to_dataframe(self):
- results = []
- cache_manager = ie.current_env().get_cache_manager(self._pcoll.pipeline)
- if cache_manager.exists('full', self._cache_key):
- coder = cache_manager.load_pcoder('full', self._cache_key)
- reader, _ = cache_manager.read('full', self._cache_key)
- results = list(to_element_list(reader, coder, include_window_info=True))
-
Review comment:
Probably known issue https://issues.apache.org/jira/browse/BEAM-10627, an AI is https://issues.apache.org/jira/browse/BEAM-10514. It's possible the cache path exceeded the windows default path length limit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689048887
# [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=h1) Report
> Merging [#12704](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2bb60c323095340240ec4982c1e9dabc397107e5?el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `16.80%`.
[](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
==========================================
- Coverage 40.43% 40.40% -0.04%
==========================================
Files 449 451 +2
Lines 53530 53685 +155
==========================================
+ Hits 21643 21689 +46
- Misses 31887 31996 +109
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `28.57% <0.00%> (-0.19%)` | :arrow_down: |
| [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `20.20% <0.00%> (ø)` | |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `27.92% <0.00%> (-1.14%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `28.06% <0.00%> (+0.04%)` | :arrow_up: |
| [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `34.61% <6.97%> (+5.37%)` | :arrow_up: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `24.34% <9.09%> (-0.66%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `24.21% <12.50%> (-0.55%)` | :arrow_down: |
| ... and [5 more](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=footer). Last update [1463ff0...79a72ba](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #12704:
URL: https://github.com/apache/beam/pull/12704
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on a change in pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #12704:
URL: https://github.com/apache/beam/pull/12704#discussion_r486535437
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
##########
@@ -342,121 +343,79 @@ def show(*pcolls, **configs):
assert isinstance(pcoll, beam.pvalue.PCollection), (
'{} is not an apache_beam.pvalue.PCollection.'.format(pcoll))
user_pipeline = pcolls[0].pipeline
- for pcoll in pcolls:
- assert pcoll.pipeline is user_pipeline, (
- '{} belongs to a different user-defined pipeline ({}) than that of'
- ' other PCollections ({}).'.format(
- pcoll, pcoll.pipeline, user_pipeline))
+
# TODO(BEAM-8288): Remove below pops and assertion once Python 2 is
# deprecated from Beam.
include_window_info = configs.pop('include_window_info', False)
visualize_data = configs.pop('visualize_data', False)
+ n = configs.pop('n', 'inf')
+ duration = configs.pop('duration', 'inf')
+
+ if n == 'inf':
+ n = float('inf')
+
+ if duration == 'inf':
+ duration = float('inf')
+
# This assertion is to protect the backward compatibility for function
# signature change after Python 2 deprecation.
assert not configs, (
'The only configs supported are include_window_info and '
'visualize_data.')
- runner = user_pipeline.runner
- if isinstance(runner, ir.InteractiveRunner):
- runner = runner._underlying_runner
-
- # Make sure that sources without a user reference are still cached.
- pi.watch_sources(user_pipeline)
-
- # Make sure that all PCollections to be shown are watched. If a PCollection
- # has not been watched, make up a variable name for that PCollection and watch
- # it. No validation is needed here because the watch logic can handle
- # arbitrary variables.
- watched_pcollections = set()
- for watching in ie.current_env().watching():
- for _, val in watching:
- if hasattr(val, '__class__') and isinstance(val, beam.pvalue.PCollection):
- watched_pcollections.add(val)
- for pcoll in pcolls:
- if pcoll not in watched_pcollections:
- watch({'anonymous_pcollection_{}'.format(id(pcoll)): pcoll})
-
- if ie.current_env().is_in_ipython:
- warnings.filterwarnings(
- 'ignore',
- 'options is deprecated since First stable release. References to '
- '<pipeline>.options will not be supported',
- category=DeprecationWarning)
- # Attempt to run background caching job since we have the reference to the
- # user-defined pipeline.
- bcj.attempt_to_run_background_caching_job(
- runner, user_pipeline, user_pipeline.options)
-
- pcolls = set(pcolls)
- computed_pcolls = set()
- for pcoll in pcolls:
- if pcoll in ie.current_env().computed_pcollections:
- computed_pcolls.add(pcoll)
- pcolls = pcolls.difference(computed_pcolls)
- # If in notebook, static plotting computed pcolls as computation is done.
- if ie.current_env().is_in_notebook:
- for pcoll in computed_pcolls:
- visualize(
- pcoll,
- include_window_info=include_window_info,
- display_facets=visualize_data)
- elif ie.current_env().is_in_ipython:
- for pcoll in computed_pcolls:
- visualize(pcoll, include_window_info=include_window_info)
-
- if not pcolls:
- return
-
- # Build a pipeline fragment for the PCollections and run it.
- result = pf.PipelineFragment(list(pcolls), user_pipeline.options).run()
- ie.current_env().set_pipeline_result(user_pipeline, result)
-
- # If in notebook, dynamic plotting as computation goes.
- if ie.current_env().is_in_notebook:
- for pcoll in pcolls:
- visualize(
- pcoll,
- dynamic_plotting_interval=1,
- include_window_info=include_window_info,
- display_facets=visualize_data)
-
- # Invoke wait_until_finish to ensure the blocking nature of this API without
- # relying on the run to be blocking.
- result.wait_until_finish()
-
- # If just in ipython shell, plotting once when the computation is completed.
- if ie.current_env().is_in_ipython and not ie.current_env().is_in_notebook:
- for pcoll in pcolls:
- visualize(pcoll, include_window_info=include_window_info)
-
- # If the pipeline execution is successful at this stage, mark the computation
- # completeness for the given PCollections so that when further `show`
- # invocation occurs, Interactive Beam wouldn't need to re-compute them.
- if result.state is beam.runners.runner.PipelineState.DONE:
- ie.current_env().mark_pcollection_computed(pcolls)
-
-
-def collect(pcoll, include_window_info=False):
- """Materializes all of the elements from a PCollection into a Dataframe.
- For example::
+ recording_manager = RecordingManager(user_pipeline)
+ recording = recording_manager.record(
+ pcolls, max_n=n, max_duration_secs=duration)
+
+ # Catch a KeyboardInterrupt to gracefully cancel the recording and
+ # visualizations.
+ try:
+ # If in notebook, static plotting computed pcolls as computation is done.
+ if ie.current_env().is_in_notebook:
+ for stream in recording.computed().values():
+ visualize(
+ stream,
+ include_window_info=include_window_info,
+ display_facets=visualize_data)
+ elif ie.current_env().is_in_ipython:
+ for stream in recording.computed().values():
+ visualize(stream, include_window_info=include_window_info)
Review comment:
Yep, I have verified it. It uses the same as the code that it was replacing by using the cache key.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689044025
R: @KevinGG
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689048887
# [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=h1) Report
> Merging [#12704](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/b45a1b6a10de795c07016bc4d5b1c5cca76b3624?el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `74.75%`.
[](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
=======================================
Coverage 82.38% 82.38%
=======================================
Files 451 451
Lines 53721 53713 -8
=======================================
- Hits 44256 44251 -5
+ Misses 9465 9462 -3
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `98.05% <84.61%> (+3.45%)` | :arrow_up: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `94.78% <90.90%> (-1.38%)` | :arrow_down: |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-0.46%)` | :arrow_down: |
| [...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==) | `89.74% <0.00%> (+0.10%)` | :arrow_up: |
| [.../runners/portability/fn\_api\_runner/translations.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3RyYW5zbGF0aW9ucy5weQ==) | `88.17% <0.00%> (+0.11%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.34% <0.00%> (+0.35%)` | :arrow_up: |
| ... and [5 more](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=footer). Last update [1463ff0...71d4a90](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689048887
# [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=h1) Report
> Merging [#12704](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2bb60c323095340240ec4982c1e9dabc397107e5?el=desc) will **decrease** coverage by `0.14%`.
> The diff coverage is `11.62%`.
[](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
==========================================
- Coverage 40.43% 40.28% -0.15%
==========================================
Files 449 455 +6
Lines 53530 53792 +262
==========================================
+ Hits 21643 21670 +27
- Misses 31887 32122 +235
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...am/runners/interactive/options/capture\_limiters.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfbGltaXRlcnMucHk=) | `40.32% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `34.61% <6.97%> (+5.37%)` | :arrow_up: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `24.21% <12.50%> (-0.55%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `28.08% <15.62%> (-0.98%)` | :arrow_down: |
| [...he\_beam/runners/interactive/pipeline\_instrument.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9waXBlbGluZV9pbnN0cnVtZW50LnB5) | `14.86% <50.00%> (ø)` | |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `24.23% <0.00%> (-0.05%)` | :arrow_down: |
| [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `20.20% <0.00%> (ø)` | |
| [...thon/apache\_beam/tools/sideinput\_microbenchmark.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdG9vbHMvc2lkZWlucHV0X21pY3JvYmVuY2htYXJrLnB5) | `0.00% <0.00%> (ø)` | |
| ... and [5 more](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=footer). Last update [1463ff0...d78dee0](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-690537687
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-694379028
It looks like this made the windows tests in Python very flaky, I don't think all three have passed in any runs since (BEAM-10921).
I tested in a rollback (https://github.com/apache/beam/pull/12865) and all three passed again. I think we should merge that rollback while the flakiness is debugged, unless there's a clear fix?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12704: [BEAM-10603] Implement the new Large Source Recording API.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12704:
URL: https://github.com/apache/beam/pull/12704#issuecomment-689048887
# [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=h1) Report
> Merging [#12704](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2bb60c323095340240ec4982c1e9dabc397107e5?el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `16.80%`.
[](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
==========================================
- Coverage 40.43% 40.40% -0.04%
==========================================
Files 449 451 +2
Lines 53530 53685 +155
==========================================
+ Hits 21643 21689 +46
- Misses 31887 31996 +109
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `28.57% <0.00%> (-0.19%)` | :arrow_down: |
| [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `20.20% <0.00%> (ø)` | |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `27.92% <0.00%> (-1.14%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `28.06% <0.00%> (+0.04%)` | :arrow_up: |
| [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_perf.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19wZXJmLnB5) | `0.00% <0.00%> (ø)` | |
| [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `34.61% <6.97%> (+5.37%)` | :arrow_up: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `24.34% <9.09%> (-0.66%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `24.21% <12.50%> (-0.55%)` | :arrow_down: |
| ... and [5 more](https://codecov.io/gh/apache/beam/pull/12704/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=footer). Last update [1463ff0...d78dee0](https://codecov.io/gh/apache/beam/pull/12704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org