You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Polber (via GitHub)" <gi...@apache.org> on 2023/09/07 22:24:50 UTC
[GitHub] [beam] Polber opened a new pull request, #28363: Jkinard/add udf docs
Polber opened a new pull request, #28363:
URL: https://github.com/apache/beam/pull/28363
PR adds docs for UDF's to `yaml_mapping.md` and moves the UDF tests to `yaml_mapping_test.py`.
Requires go/beam-pr/28114 be merged first.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [x] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #28363: [yaml] Add UDF docs
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-1711700302
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28363?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#28363](https://app.codecov.io/gh/apache/beam/pull/28363?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (fdde3d4) into [master](https://app.codecov.io/gh/apache/beam/commit/5cc87092b3f70581f8351700136fa229cf4ce299?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (5cc8709) will **increase** coverage by `0.00%`.
> Report is 10 commits behind head on master.
> The diff coverage is `84.41%`.
```diff
@@ Coverage Diff @@
## master #28363 +/- ##
=======================================
Coverage 72.33% 72.34%
=======================================
Files 681 681
Lines 100355 100409 +54
=======================================
+ Hits 72593 72641 +48
- Misses 26185 26191 +6
Partials 1577 1577
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/28363/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/apache/beam/pull/28363/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.90% <84.41%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/beam/pull/28363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [sdks/python/setup.py](https://app.codecov.io/gh/apache/beam/pull/28363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vc2V0dXAucHk=) | `0.00% <ø> (ø)` | |
| [...on/apache\_beam/ml/inference/vertex\_ai\_inference.py](https://app.codecov.io/gh/apache/beam/pull/28363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL3ZlcnRleF9haV9pbmZlcmVuY2UucHk=) | `44.92% <50.00%> (+0.15%)` | :arrow_up: |
| [sdks/python/apache\_beam/yaml/yaml\_mapping.py](https://app.codecov.io/gh/apache/beam/pull/28363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0veWFtbC95YW1sX21hcHBpbmcucHk=) | `86.16% <83.87%> (-0.80%)` | :arrow_down: |
| [sdks/python/apache\_beam/utils/python\_callable.py](https://app.codecov.io/gh/apache/beam/pull/28363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHl0aG9uX2NhbGxhYmxlLnB5) | `87.09% <92.30%> (+0.21%)` | :arrow_up: |
... and [5 files with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/28363/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [yaml] Add UDF docs [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-2543860038
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [yaml] Add UDF docs [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-1783047986
@Polber what are next steps here? Can we clean up the conflicts and merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [yaml] Add UDF docs [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-1937697319
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [yaml] Add UDF docs [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-1783053619
@damccorm I have another PR I want to get in before refactoring this. Should open that one today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [yaml] Add UDF docs [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-2411136620
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [yaml] Add UDF docs [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber closed pull request #28363: [yaml] Add UDF docs
URL: https://github.com/apache/beam/pull/28363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [yaml] Add UDF docs [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-1852079307
@Polber what is the plan for this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #28363: [yaml] Add UDF docs
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-1710866214
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #28363: [yaml] Add UDF docs
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #28363:
URL: https://github.com/apache/beam/pull/28363#discussion_r1320349719
##########
sdks/python/apache_beam/yaml/readme_test.py:
##########
@@ -155,14 +161,27 @@ def __exit__(self, *args):
self.tempdir.cleanup()
+def replace_mapping_fields(spec, arg_name, arg_value):
+ if arg_name != 'language' and arg_name != 'fields':
+ spec['config']['fields'] = {
+ k: {arg: arg_value if arg == arg_name else v[arg]
+ for arg in v} if isinstance(v, dict) else v
+ for k,
Review Comment:
Yapf messes up on `k, v`. You can put in ()'s for better wrapping.
##########
sdks/python/apache_beam/yaml/readme_test.py:
##########
@@ -143,6 +143,12 @@ def input_file(self, name, content):
def input_csv(self):
return self.input_file('input.csv', 'col1,col2,col3\nabc,1,2.5\n')
+ def input_udf(self):
+ return self.input_file(
+ 'udf.py',
+ 'def my_mapping(row):\n return row.col2 > 0\n'
Review Comment:
Maybe use triple quotes for this content so it's easier to read.
##########
sdks/python/apache_beam/yaml/readme_test.py:
##########
@@ -233,9 +254,9 @@ def parse_test_methods(markdown_lines):
if 'providers:' in yaml_pipeline:
test_type = 'PARSE'
yield test_name, create_test_method(
- test_type,
- test_name,
- yaml_pipeline)
+ test_type,
Review Comment:
This was correct before. Check your editor settings?
##########
sdks/python/apache_beam/yaml/readme_test.py:
##########
@@ -155,14 +161,27 @@ def __exit__(self, *args):
self.tempdir.cleanup()
+def replace_mapping_fields(spec, arg_name, arg_value):
+ if arg_name != 'language' and arg_name != 'fields':
Review Comment:
Can this be phrased positively? I don't know what `arg_name` could be here. Is `language/fields` ever actually passed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [yaml] Add UDF docs [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-2167964268
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [yaml] Add UDF docs [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-2288641840
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [yaml] Add UDF docs [beam]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-2053635491
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28363?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 72.24%. Comparing base [(`6261a00`)](https://app.codecov.io/gh/apache/beam/commit/6261a007a9e1f3eff2ba7cc003d96bd5d77231a9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`9518d90`)](https://app.codecov.io/gh/apache/beam/pull/28363?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1645 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #28363 +/- ##
=======================================
Coverage 72.24% 72.24%
=======================================
Files 684 684
Lines 100946 100961 +15
=======================================
+ Hits 72924 72943 +19
+ Misses 26443 26439 -4
Partials 1579 1579
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/28363/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/apache/beam/pull/28363/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.84% <ø> (+0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/28363?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [yaml] Add UDF docs [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-2053634934
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Polber commented on pull request #28363: [yaml] Add UDF docs
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on PR #28363:
URL: https://github.com/apache/beam/pull/28363#issuecomment-1710865431
R: @robertwb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org