You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/06 21:59:07 UTC
[GitHub] [beam] epicfaace opened a new pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
epicfaace opened a new pull request #12779:
URL: https://github.com/apache/beam/pull/12779
Support for NestedValueProvider for Python SDK. I've also added more comments to the pydoc for value_provider classes. R: @pabloem
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691339688
@epicfaace Curious, what use-case do you have in mind for this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] arvindram03 commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
arvindram03 commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691354649
Hi @epicfaace ,
We agree the startup time with flex template python jobs is an issue. We are are working towards reducing the overall startup time to 3-4 mins by end of this year. We recommend using flex templates instead of classic ones purely for flexibility and also you can leverage more feature expansions in the near future.
One of the major objective of flex templates is to avoid using ValueProviders. So, further overloading them with new features is not aligned with the roadmap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-692856347
# [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=h1) Report
> Merging [#12779](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=desc) (af2c14c) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **decrease** coverage by `0.20%`.
> The diff coverage is `93.33%`.
[](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12779 +/- ##
==========================================
- Coverage 82.48% 82.28% -0.21%
==========================================
Files 455 451 -4
Lines 54876 53738 -1138
==========================================
- Hits 45266 44217 -1049
+ Misses 9610 9521 -89
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/options/value\_provider.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy92YWx1ZV9wcm92aWRlci5weQ==) | `91.76% <93.33%> (+0.21%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (-54.35%)` | :arrow_down: |
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `83.33% <0.00%> (-9.53%)` | :arrow_down: |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: |
| [conftest.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-Y29uZnRlc3QucHk=) | `77.77% <0.00%> (-7.94%)` | :arrow_down: |
| [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | `87.36% <0.00%> (-7.37%)` | :arrow_down: |
| [sdks/python/apache\_beam/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vX19pbml0X18ucHk=) | `80.00% <0.00%> (-5.72%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/partitionings.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhcnRpdGlvbmluZ3MucHk=) | `83.60% <0.00%> (-5.44%)` | :arrow_down: |
| [.../python/apache\_beam/io/gcp/bigquery\_io\_metadata.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2lvX21ldGFkYXRhLnB5) | `86.95% <0.00%> (-3.67%)` | :arrow_down: |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <0.00%> (-3.51%)` | :arrow_down: |
| ... and [87 more](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=footer). Last update [1a34854...912e912](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691350944
Hmm, I did try using Flex Templates initially, but it took way too long -- maybe 7-8 minutes -- for the job to initialize, as I think it was installing apache-beam and other dependencies. I switched to using ValueProviders because I needed a quick turnaround time for the job (the job itself takes around 10 minutes, so a 7-8 minute additional startup time is not worth it for me).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691339409
https://cwiki.apache.org/confluence/display/BEAM/Python+Tips has some autoformatter tips that can help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691350944
Hmm, I did try using Flex Templates initially, but it took way too long -- maybe 7-8 minutes -- for the job to initialize, and it appears that the bottleneck is with was installing apache-beam and other pip dependencies. I switched to using ValueProviders because I needed a quick turnaround time for the job (the job itself takes around 10 minutes, so a 7-8 minute additional startup time is not worth it for me).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #12779:
URL: https://github.com/apache/beam/pull/12779
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-731751916
@pabloem I'm not quite sure why CI is still failing, it gives this error:
```
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-4m1c1oje2145793178144\\full\\fb91a47796-2145832985040-2145832986608-2145793178144'
```
Any ideas? Or is the CI just flaky?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] arvindram03 commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
arvindram03 commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691354649
Hi @epicfaace ,
We agree the startup time with flex template python jobs is an issue. We are are working towards reducing the overall startup time to 3-4 mins by end of this year. We recommend using flex templates instead of classic ones purely for flexibility and also you can leverage more feature expansions in the near future.
One of the major objective of flex templates is to avoid using ValueProviders. So, further overloading them with new features is not aligned with the roadmap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
tvalentyn edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691350351
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nikie commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
nikie commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-731390355
@pabloem, Python Docs and Lint tests now succeed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-732535245
> > BeamAssertException
>
> This one looks like a test on the asserts.
>
> The actual error (which is likely a flake) is:
>
> logs: https://github.com/apache/beam/pull/12779/checks?check_run_id=1433935136
>
> ```
> apache_beam\runners\interactive\recording_manager_test.py:75:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> apache_beam\runners\interactive\interactive_environment.py:118: in new_env
> _interactive_beam_env.cleanup()
> apache_beam\runners\interactive\interactive_environment.py:272: in cleanup
> cache_manager.cleanup()
> apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
> shutil.rmtree(self._cache_dir)
> c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:500: in rmtree
> return _rmtree_unsafe(path, onerror)
> c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:390: in _rmtree_unsafe
> _rmtree_unsafe(fullname, onerror)
> c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:395: in _rmtree_unsafe
> onerror(os.unlink, fullname, sys.exc_info())
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> ...
>
> > os.unlink(fullname)
> E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-4m1c1oje2145793178144\\full\\fb91a47796-2145832985040-2145832986608-2145793178144'
> ```
>
> Maybe clean should catch the error and not fail? Or try again?
>
> @KevinGG - Could you check this error?
The problem is windows specific. I tried on Mac OS, the `shutil.rmtree` works even if the file handler is still open:
<img width="964" alt="Screen Shot 2020-11-23 at 5 11 51 PM" src="https://user-images.githubusercontent.com/4423149/100034273-b343ef80-2db0-11eb-8988-dcfc2c79322c.png">
It's said that on windows, if a file is opened, even if in the same process deleting it (or its parent directory), you will see this error: https://stackoverflow.com/questions/27215462/permissionerror-winerror-32-the-process-cannot-access-the-file-because-it-is.
Potential breaking point is in the streaming cache where the `open` and `close` are explicitly invoked (where `with` is inappropriate to use):
<img width="621" alt="Screen Shot 2020-11-23 at 5 44 53 PM" src="https://user-images.githubusercontent.com/4423149/100035692-a7a5f800-2db3-11eb-8ec8-abc8923406c0.png">
Is it possible that a `finish_bundle` would fail silently or not executed at all?
This probably does not happen that often, we can revisit this if it happens frequently and ignore this by specifying `shutil.rmtree(path, ignore_errors=True, onerror=...)` since it's just failing to clean up useless temp files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-692856347
# [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=h1) Report
> Merging [#12779](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/a2126e571cfddc90e3819e66ef1f2da3d52833ed?el=desc) will **decrease** coverage by `0.08%`.
> The diff coverage is `44.11%`.
[](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12779 +/- ##
==========================================
- Coverage 82.36% 82.28% -0.09%
==========================================
Files 450 451 +1
Lines 53708 53738 +30
==========================================
- Hits 44238 44217 -21
- Misses 9470 9521 +51
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/options/value\_provider.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy92YWx1ZV9wcm92aWRlci5weQ==) | `91.76% <93.33%> (+0.21%)` | :arrow_up: |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.45% <0.00%> (-0.14%)` | :arrow_down: |
| [sdks/python/apache\_beam/pvalue.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcHZhbHVlLnB5) | `92.01% <0.00%> (+0.38%)` | :arrow_up: |
| ... and [5 more](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=footer). Last update [5206131...af2c14c](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-692856347
# [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=h1) Report
> Merging [#12779](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/a2126e571cfddc90e3819e66ef1f2da3d52833ed?el=desc) will **decrease** coverage by `0.08%`.
> The diff coverage is `44.11%`.
[](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12779 +/- ##
==========================================
- Coverage 82.36% 82.28% -0.09%
==========================================
Files 450 451 +1
Lines 53708 53738 +30
==========================================
- Hits 44238 44217 -21
- Misses 9470 9521 +51
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/options/value\_provider.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy92YWx1ZV9wcm92aWRlci5weQ==) | `91.76% <93.33%> (+0.21%)` | :arrow_up: |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.98% <0.00%> (-0.36%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.45% <0.00%> (-0.14%)` | :arrow_down: |
| [sdks/python/apache\_beam/pvalue.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcHZhbHVlLnB5) | `92.01% <0.00%> (+0.38%)` | :arrow_up: |
| ... and [5 more](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=footer). Last update [5206131...af2c14c](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nikie commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
nikie commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-727594594
@tvalentyn
Another possible use case for `NestedValueProvider` is to fetch external secrets based on provided values from, for example, Vault.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-736696949
thank you @epicfaace !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-705755411
What is the next step on this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-705755411
What is the next step on this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-705760739
@epicfaace there's a number of failing tests. (Python Docs and Lint) - can you fix those?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-705760739
@epicfaace there's a number of failing tests. (Python Docs and Lint) - can you fix those?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691352337
I see, I'd like @arvindram03 and @azurezyq chime in here on their recommendation. I think there are undergoing efforts to reduce the startup time, or perhaps there are some tips to reduce it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-692856347
# [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=h1) Report
> Merging [#12779](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=desc) (af2c14c) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **decrease** coverage by `0.20%`.
> The diff coverage is `93.33%`.
[](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12779 +/- ##
==========================================
- Coverage 82.48% 82.28% -0.21%
==========================================
Files 455 451 -4
Lines 54876 53738 -1138
==========================================
- Hits 45266 44217 -1049
+ Misses 9610 9521 -89
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/options/value\_provider.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy92YWx1ZV9wcm92aWRlci5weQ==) | `91.76% <93.33%> (+0.21%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (-54.35%)` | :arrow_down: |
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `83.33% <0.00%> (-9.53%)` | :arrow_down: |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: |
| [conftest.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-Y29uZnRlc3QucHk=) | `77.77% <0.00%> (-7.94%)` | :arrow_down: |
| [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | `87.36% <0.00%> (-7.37%)` | :arrow_down: |
| [sdks/python/apache\_beam/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vX19pbml0X18ucHk=) | `80.00% <0.00%> (-5.72%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/partitionings.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhcnRpdGlvbmluZ3MucHk=) | `83.60% <0.00%> (-5.44%)` | :arrow_down: |
| [.../python/apache\_beam/io/gcp/bigquery\_io\_metadata.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2lvX21ldGFkYXRhLnB5) | `86.95% <0.00%> (-3.67%)` | :arrow_down: |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <0.00%> (-3.51%)` | :arrow_down: |
| ... and [87 more](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=footer). Last update [eba648c...0f7961d](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691339409
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691342588
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
tvalentyn edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691350351
Ok. We actually consider ValueProvider's an anti-pattern now, prefer not to extend their scope, and recommend to use FlexTemplates instead, see: https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates. Could you give that a try? I do appreciate your effort to improve the SDK and documentation though!
CC: @azurezyq who is an expert on Flex Templates.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691350944
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-692856347
# [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=h1) Report
> Merging [#12779](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) will **decrease** coverage by `0.07%`.
> The diff coverage is `51.53%`.
[](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12779 +/- ##
==========================================
- Coverage 82.35% 82.28% -0.08%
==========================================
Files 450 451 +1
Lines 53708 53738 +30
==========================================
- Hits 44230 44217 -13
- Misses 9478 9521 +43
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <85.71%> (ø)` | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/options/value\_provider.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy92YWx1ZV9wcm92aWRlci5weQ==) | `91.76% <93.33%> (+0.21%)` | :arrow_up: |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <100.00%> (ø)` | |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <100.00%> (ø)` | |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: |
| ... and [14 more](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=footer). Last update [5206131...af2c14c](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-731751916
@pabloem I'm not quite sure why CI is still failing, it gives this error:
```
return self.do_fn_invoker.invoke_process(windowed_value)
File "D:\a\beam\beam\sdks\python\apache_beam\runners\common.py", line 570, in invoke_process
windowed_value, self.process_method(windowed_value.value))
File "D:\a\beam\beam\sdks\python\apache_beam\runners\common.py", line 1374, in process_outputs
self.main_receivers.receive(windowed_value)
File "D:\a\beam\beam\sdks\python\apache_beam\runners\worker\operations.py", line 220, in receive
self.consumer.process(windowed_value)
File "D:\a\beam\beam\sdks\python\apache_beam\runners\worker\operations.py", line 718, in process
delayed_application = self.dofn_runner.process(o)
File "D:\a\beam\beam\sdks\python\apache_beam\runners\common.py", line 1215, in process
self._reraise_augmented(exn)
File "D:\a\beam\beam\sdks\python\apache_beam\runners\common.py", line 1294, in _reraise_augmented
raise_with_traceback(new_exn)
File "d:\a\beam\beam\sdks\python\target\.tox\py36-win\lib\site-packages\future\utils\__init__.py", line 446, in raise_with_traceback
raise exc.with_traceback(traceback)
File "D:\a\beam\beam\sdks\python\apache_beam\runners\common.py", line 1213, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File "D:\a\beam\beam\sdks\python\apache_beam\runners\common.py", line 743, in invoke_process
windowed_value, additional_args, additional_kwargs)
File "D:\a\beam\beam\sdks\python\apache_beam\runners\common.py", line 867, in _invoke_process_per_window
self.process_method(*args_for_process),
File "D:\a\beam\beam\sdks\python\apache_beam\transforms\core.py", line 1669, in <lambda>
wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)]
File "D:\a\beam\beam\sdks\python\apache_beam\testing\util.py", line 197, in _equal
raise BeamAssertException(msg)
apache_beam.testing.util.BeamAssertException: Failed assert: ['a'] == ['a', 'b'], unexpected elements ['b'] [while running 'assert_that/Match']
```
Any ideas? Or is the CI just flaky?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-692856347
# [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=h1) Report
> Merging [#12779](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=desc) (af2c14c) into [master](https://codecov.io/gh/apache/beam/commit/e0e5fb529d3c7184151142c3c9bbc23ed3b20644?el=desc) (e0e5fb5) will **decrease** coverage by `0.07%`.
> The diff coverage is `51.53%`.
[](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12779 +/- ##
==========================================
- Coverage 82.35% 82.28% -0.08%
==========================================
Files 450 451 +1
Lines 53708 53738 +30
==========================================
- Hits 44230 44217 -13
- Misses 9478 9521 +43
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...beam/testing/benchmarks/nexmark/queries/query10.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTEwLnB5) | `0.00% <0.00%> (ø)` | |
| [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.06% <50.00%> (ø)` | |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <64.40%> (-10.53%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.00% <85.71%> (ø)` | |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `85.26% <87.50%> (-0.88%)` | :arrow_down: |
| [sdks/python/apache\_beam/options/value\_provider.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy92YWx1ZV9wcm92aWRlci5weQ==) | `91.76% <93.33%> (+0.21%)` | :arrow_up: |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `88.83% <100.00%> (ø)` | |
| [...python/apache\_beam/runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `93.70% <100.00%> (ø)` | |
| [...ive/messaging/interactive\_environment\_inspector.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9tZXNzYWdpbmcvaW50ZXJhY3RpdmVfZW52aXJvbm1lbnRfaW5zcGVjdG9yLnB5) | `97.43% <100.00%> (ø)` | |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: |
| ... and [14 more](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=footer). Last update [5206131...006a79f](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-692833522
> A user has written a feature that they would find useful, and that will not change the experience for other users (if anything, it should improve it). The feature looks correct, and similar to what we do in Java. If we reject the PR, we may push the user to run on a fork. Can we let this in? @tvalentyn
I agree with this assessment, feel free to merge once tests pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691342588
> @epicfaace Curious, what use-case do you have in mind for this change?
I'm creating a Dataflow template in which I need to write to multiple different output tables, "[prefix]_a", "[prefix]_b", "[prefix]_c", etc. I'd like to be able to specify a single runtime parameter, "prefix", and derive the output table names from this parameter, rather than having to redundantly specify each table name as an input runtime parameter.
I noticed that [this page](https://cloud.google.com/dataflow/docs/guides/templates/creating-templates#using-nestedvalueprovider) said that the Apache Beam SDK for Python does not support NestedValueProvider, so I just implemented it in Python in a similar way to how the Java SDK does it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-689251584
By the way, how do you normally lint your code according to the Beam guidelines / PR checks? I couldn't find any guidance / a quick reference in the Beam development guide on how to do that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-693433346
@epicfaace can you ensure PythonDocs and PythonLint checks pass?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-692107896
A user has written a feature that they would find useful, and that will not change the experience for other users (if anything, it should improve it). The feature looks correct, and similar to what we do in Java. If we reject the PR, we may push the user to run on a fork. Can we let this in? @tvalentyn
Flex templates have numerous benefits that are not being undone or discounted by adding this improvement to traditional templates, so I don't see a great disadvantage to merge this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-701049001
@epicfaace LMK if you can take a look at the lint issues
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-692856347
# [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=h1) Report
> Merging [#12779](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=desc) (af2c14c) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **decrease** coverage by `0.20%`.
> The diff coverage is `93.33%`.
[](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12779 +/- ##
==========================================
- Coverage 82.48% 82.28% -0.21%
==========================================
Files 455 451 -4
Lines 54876 53738 -1138
==========================================
- Hits 45266 44217 -1049
+ Misses 9610 9521 -89
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/options/value\_provider.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy92YWx1ZV9wcm92aWRlci5weQ==) | `91.76% <93.33%> (+0.21%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (-54.35%)` | :arrow_down: |
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `83.33% <0.00%> (-9.53%)` | :arrow_down: |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: |
| [conftest.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-Y29uZnRlc3QucHk=) | `77.77% <0.00%> (-7.94%)` | :arrow_down: |
| [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | `87.36% <0.00%> (-7.37%)` | :arrow_down: |
| [sdks/python/apache\_beam/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vX19pbml0X18ucHk=) | `80.00% <0.00%> (-5.72%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/partitionings.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhcnRpdGlvbmluZ3MucHk=) | `83.60% <0.00%> (-5.44%)` | :arrow_down: |
| [.../python/apache\_beam/io/gcp/bigquery\_io\_metadata.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2lvX21ldGFkYXRhLnB5) | `86.95% <0.00%> (-3.67%)` | :arrow_down: |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <0.00%> (-3.51%)` | :arrow_down: |
| ... and [87 more](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=footer). Last update [eba648c...c455e46](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-688867355
Thanks @epicfaace for implementing these! I will not be able to review them for a couple of weeks. I can review once I'm back, or I can help find a new reviewer. What do you prefer?
(tvalentyn, udim) would be good choices for reviewers if you'd like a review soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] arvindram03 commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
arvindram03 commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691354649
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-689250939
Thanks for the heads up @pabloem . @tvalentyn , would you be able to review?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] arvindram03 commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
arvindram03 commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691354649
Hi @epicfaace ,
We agree the startup time with flex template python jobs is an issue. We are are working towards reducing the overall startup time to 3-4 mins by end of this year. We recommend using flex templates instead of classic ones purely for flexibility and also you can leverage more feature expansions in the near future.
One of the major objective of flex templates is to avoid using ValueProviders. So, further overloading them with new features is not aligned with the roadmap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nikie commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
nikie commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-727593345
@epicfaace @pabloem
I have open PR into @epicfaace 's branch to fix pylint errors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-732610284
> > > BeamAssertException
> >
> >
> > This one looks like a test on the asserts.
> > The actual error (which is likely a flake) is:
> > logs: [#12779 (checks)](https://github.com/apache/beam/pull/12779/checks?check_run_id=1433935136)
> > ```
> > apache_beam\runners\interactive\recording_manager_test.py:75:
> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> > apache_beam\runners\interactive\interactive_environment.py:118: in new_env
> > _interactive_beam_env.cleanup()
> > apache_beam\runners\interactive\interactive_environment.py:272: in cleanup
> > cache_manager.cleanup()
> > apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
> > shutil.rmtree(self._cache_dir)
> > c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:500: in rmtree
> > return _rmtree_unsafe(path, onerror)
> > c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:390: in _rmtree_unsafe
> > _rmtree_unsafe(fullname, onerror)
> > c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:395: in _rmtree_unsafe
> > onerror(os.unlink, fullname, sys.exc_info())
> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> >
> > ...
> >
> > > os.unlink(fullname)
> > E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-4m1c1oje2145793178144\\full\\fb91a47796-2145832985040-2145832986608-2145793178144'
> > ```
> >
> >
> > Maybe clean should catch the error and not fail? Or try again?
> > @KevinGG - Could you check this error?
>
> The problem is windows specific. I tried on Mac OS, the `shutil.rmtree` works even if the file handler is still open:
> <img alt="Screen Shot 2020-11-23 at 5 11 51 PM" width="964" src="https://user-images.githubusercontent.com/4423149/100034273-b343ef80-2db0-11eb-8988-dcfc2c79322c.png">
> It's said that on windows, if a file is opened, even if in the same process deleting it (or its parent directory), you will see this error: [stackoverflow.com/questions/27215462/permissionerror-winerror-32-the-process-cannot-access-the-file-because-it-is](https://stackoverflow.com/questions/27215462/permissionerror-winerror-32-the-process-cannot-access-the-file-because-it-is).
>
> Potential breaking point is in the streaming cache where the `open` and `close` are explicitly invoked (where `with` is inappropriate to use):
> <img alt="Screen Shot 2020-11-23 at 5 44 53 PM" width="621" src="https://user-images.githubusercontent.com/4423149/100035692-a7a5f800-2db3-11eb-8ec8-abc8923406c0.png">
> Is it possible that a `finish_bundle` would fail silently or not executed at all?
>
> This probably does not happen that often, we can revisit this if it happens frequently and ignore this by specifying `shutil.rmtree(path, ignore_errors=True, onerror=...)` since it's just failing to clean up useless temp files.
Maybe file a JIRA and we can track how often this happens? I do not think people triage all failures so it is hard to tell how frequently this happens in pre commit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691339409
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
aaltay edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-732388291
> BeamAssertException
This one looks like a test on the asserts.
The actual error (which is likely a flake) is:
logs: https://github.com/apache/beam/pull/12779/checks?check_run_id=1433935136
```
apache_beam\runners\interactive\recording_manager_test.py:75:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam\runners\interactive\interactive_environment.py:118: in new_env
_interactive_beam_env.cleanup()
apache_beam\runners\interactive\interactive_environment.py:272: in cleanup
cache_manager.cleanup()
apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
shutil.rmtree(self._cache_dir)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:500: in rmtree
return _rmtree_unsafe(path, onerror)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:390: in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:395: in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
...
> os.unlink(fullname)
E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-4m1c1oje2145793178144\\full\\fb91a47796-2145832985040-2145832986608-2145793178144'
```
Maybe clean should catch the error and not fail? Or try again?
@KevinGG - Could you check this error?
@pabloem @epicfaace - I believe you can ignore this error for the purposes of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-733147794
> Maybe file a JIRA and we can track how often this happens? I do not think people triage all failures so it is hard to tell how frequently this happens in pre commit tests.
Filed BEAM-11339 for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
tvalentyn edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-692833522
> A user has written a feature that they would find useful, and that will not change the experience for other users (if anything, it should improve it). The feature looks correct, and similar to what we do in Java. If we reject the PR, we may push the user to run on a fork. Can we let this in? @tvalentyn
I agree with this assessment, feel free to merge once tests & linter pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nikie edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
nikie edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-727593345
@epicfaace @pabloem
I have opened PR into @epicfaace 's branch to fix pylint errors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-732388291
> BeamAssertException
This one looks like a test on the asserts.
The actual error (which is likely a flake) is:
logs: https://github.com/apache/beam/pull/12779/checks?check_run_id=1433935136
apache_beam\runners\interactive\recording_manager_test.py:75:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam\runners\interactive\interactive_environment.py:118: in new_env
_interactive_beam_env.cleanup()
apache_beam\runners\interactive\interactive_environment.py:272: in cleanup
cache_manager.cleanup()
apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
shutil.rmtree(self._cache_dir)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:500: in rmtree
return _rmtree_unsafe(path, onerror)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:390: in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:395: in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
...
> os.unlink(fullname)
E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-4m1c1oje2145793178144\\full\\fb91a47796-2145832985040-2145832986608-2145793178144'
Maybe clean should catch the error and not fail? Or try again?
@KevinGG - Could you check this error?
@pabloem @epicfaace - I believe you can ignore this error for the purposes of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nikie edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
nikie edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-727860809
@epicfaace
~~I am not sure why the tests are still failing.
Rebasing against current `master` might help.~~
Fixed in yet another pr: https://github.com/epicfaace/beam/pull/2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nikie commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
nikie commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-727860809
@epicfaace
I am not sure why the tests are still failing.
Rebasing against current `master` might help.
Pylint runs fine locally after my docstrings fix. I have also tried building docs with `sdks/python/scripts/generate_pydoc.sh`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
tvalentyn edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691350351
Ok. We actually consider ValueProvider's an anti-pattern now, prefer not to extend their scope, and recommend to use FlexTemplates instead, see: https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates. Could you give that a try? I do appreciate your effort to improve the SDK and documentation though!
CC: @azurezyq who is an expert on Flex Templates.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691350944
Hmm, I did try using Flex Templates initially, but it took way too long -- maybe 7-8 minutes -- for the job to initialize, and it appears that the bottleneck is with was installing apache-beam and other pip dependencies. I switched to using ValueProviders because I needed a quick turnaround time for the job (the job itself takes around 10 minutes, so a 7-8 minute additional startup time is not worth it for me).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691342588
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691350944
Hmm, I did try using Flex Templates initially, but it took way too long -- maybe 7-8 minutes -- for the job to initialize, and it appears that the bottleneck is with was installing apache-beam and other pip dependencies. I switched to using ValueProviders because I needed a quick turnaround time for the job (the job itself takes around 10 minutes, so a 7-8 minute additional startup time is not worth it for me).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691368627
@arvindram03 , thanks for the update. To clarify, this PR only aims for feature parity with the Java SDK, which already supports NestedValueProvider -- I'm not "overloading them with new features".
Additionally, Flex Templates is still a Pre-GA Offering, and as you mentioned, requires some work for it to be as efficient as regular templates. For these reasons, I don't think they're really a viable alternative to traditional templates at the moment (of course, with additional improvements by the end of the year, it could be!)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-691350351
Ok. We actually consider ValueProvider's an anti-pattern now, prefer not to extend their scope, and recommend to use FlexTemplates instead, see: https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates. Could you give that a try? I do appreciate your effort to improve the SDK and documentation though!
CC: azurezyq@ who is an expert on Flex Templates.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-692856347
# [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=h1) Report
> Merging [#12779](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=desc) (af2c14c) into [master](https://codecov.io/gh/apache/beam/commit/3d6cc0ed9ed537229b27b5dbe73288f21b0e351c?el=desc) (3d6cc0e) will **decrease** coverage by `0.20%`.
> The diff coverage is `93.33%`.
[](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12779 +/- ##
==========================================
- Coverage 82.48% 82.28% -0.21%
==========================================
Files 455 451 -4
Lines 54876 53738 -1138
==========================================
- Hits 45266 44217 -1049
+ Misses 9610 9521 -89
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/options/value\_provider.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy92YWx1ZV9wcm92aWRlci5weQ==) | `91.76% <93.33%> (+0.21%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (-54.35%)` | :arrow_down: |
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `83.33% <0.00%> (-9.53%)` | :arrow_down: |
| [...eam/runners/interactive/options/capture\_control.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9vcHRpb25zL2NhcHR1cmVfY29udHJvbC5weQ==) | `92.00% <0.00%> (-8.00%)` | :arrow_down: |
| [conftest.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-Y29uZnRlc3QucHk=) | `77.77% <0.00%> (-7.94%)` | :arrow_down: |
| [...s/snippets/transforms/aggregation/combinevalues.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9hZ2dyZWdhdGlvbi9jb21iaW5ldmFsdWVzLnB5) | `87.36% <0.00%> (-7.37%)` | :arrow_down: |
| [sdks/python/apache\_beam/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vX19pbml0X18ucHk=) | `80.00% <0.00%> (-5.72%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/partitionings.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhcnRpdGlvbmluZ3MucHk=) | `83.60% <0.00%> (-5.44%)` | :arrow_down: |
| [.../python/apache\_beam/io/gcp/bigquery\_io\_metadata.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2lvX21ldGFkYXRhLnB5) | `86.95% <0.00%> (-3.67%)` | :arrow_down: |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `76.02% <0.00%> (-3.51%)` | :arrow_down: |
| ... and [87 more](https://codecov.io/gh/apache/beam/pull/12779/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=footer). Last update [eba648c...a314678](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org