You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "bzablocki (via GitHub)" <gi...@apache.org> on 2023/05/18 19:56:21 UTC
[GitHub] [beam] bzablocki opened a new pull request, #26771: Allow for same name and type, if it is not ambiguous in yaml inputs
bzablocki opened a new pull request, #26771:
URL: https://github.com/apache/beam/pull/26771
**How it is:**
If the transform name is the same as the type, a "Value Error - Ambiguous name" is raised.
**How it should be:**
If the transform name is the same as the type, and there is no other transform of this type nor name, this is not ambiguous.
Example:
```
pipeline:
- type: ReadFromCsv
name: ReadFromCsv
path: "gs://apache-beam-samples/nasa_jpl_asteroid/sample_1000.csv"
- type: PyFilter
name: Filter
fn: "lambda astroid: (astroid.diameter or 0) > 250"
input: Read
- type: PyMap
fn: print
input: Filter
```
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #26771: Allow for same name and type, if it is not ambiguous in yaml inputs
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #26771:
URL: https://github.com/apache/beam/pull/26771#issuecomment-1553613245
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/26771?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#26771](https://app.codecov.io/gh/apache/beam/pull/26771?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (3f2f988) into [master](https://app.codecov.io/gh/apache/beam/commit/dbefe4f8b0915d4d1bd83f315705efaf0b526d3d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (dbefe4f) will **decrease** coverage by `0.06%`.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## master #26771 +/- ##
==========================================
- Coverage 72.05% 71.99% -0.06%
==========================================
Files 745 747 +2
Lines 101203 101306 +103
==========================================
+ Hits 72920 72939 +19
- Misses 26824 26908 +84
Partials 1459 1459
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `80.98% <100.00%> (-0.10%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://app.codecov.io/gh/apache/beam/pull/26771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/yaml/yaml\_transform.py](https://app.codecov.io/gh/apache/beam/pull/26771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0veWFtbC95YW1sX3RyYW5zZm9ybS5weQ==) | `86.79% <100.00%> (+0.53%)` | :arrow_up: |
... and [11 files with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/26771/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on pull request #26771: Allow for same name and type, if it is not ambiguous in yaml inputs
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #26771:
URL: https://github.com/apache/beam/pull/26771#issuecomment-1553572593
R: @robertwb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #26771: Allow for same name and type, if it is not ambiguous in yaml inputs
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #26771:
URL: https://github.com/apache/beam/pull/26771#discussion_r1198250731
##########
sdks/python/apache_beam/yaml/yaml_transform_test.py:
##########
@@ -280,6 +280,50 @@ def test_windowing_on_outer(self):
providers=TEST_PROVIDERS)
assert_that(result, equal_to([6, 9]))
+ def test_name_is_not_ambiguous(self):
+ with beam.Pipeline(options=beam.options.pipeline_options.PipelineOptions(
+ pickle_library='cloudpickle')) as p:
+
+ result = p | YamlTransform(
+ '''
+ type: composite
+ transforms:
+ - type: Create
+ name: Create
+ elements: [0, 1, 3, 4]
+ - type: PyFilter
+ name: Filter
+ keep: "lambda elem: elem > 2"
+ input: Create
+ output: Filter
+ ''')
+ # No exception raised
+ assert_that(result, equal_to([3, 4]))
+
+ def test_name_is_ambiguous(self):
+ with beam.Pipeline(options=beam.options.pipeline_options.PipelineOptions(
+ pickle_library='cloudpickle')) as p:
+
+ with self.assertRaises(ValueError):
+ result = p | YamlTransform(
+ '''
+ type: composite
+ transforms:
+ - type: Create
+ name: CreateData
+ elements: [0, 1, 3, 4]
+ - type: PyFilter
+ name: PyFilter
+ keep: "lambda elem: elem > 2"
+ input: CreateData
+ - type: PyFilter
+ name: AnotherFilter
+ keep: "lambda elem: elem > 3"
+ input: PyFilter
Review Comment:
@robertwb what do you think, should this raise an error? In my implementation it does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #26771: Allow for same name and type, if it is not ambiguous in yaml inputs
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #26771:
URL: https://github.com/apache/beam/pull/26771#issuecomment-1553808001
Looks like there's still a couple of formatting changes needed, but should be easy to fix with https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-Runningyapfformattermanually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #26771: Allow for same name and type, if it is not ambiguous in yaml inputs
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #26771:
URL: https://github.com/apache/beam/pull/26771#discussion_r1198403538
##########
sdks/python/apache_beam/yaml/yaml_transform_test.py:
##########
@@ -280,6 +280,50 @@ def test_windowing_on_outer(self):
providers=TEST_PROVIDERS)
assert_that(result, equal_to([6, 9]))
+ def test_name_is_not_ambiguous(self):
+ with beam.Pipeline(options=beam.options.pipeline_options.PipelineOptions(
+ pickle_library='cloudpickle')) as p:
+
+ result = p | YamlTransform(
+ '''
+ type: composite
+ transforms:
+ - type: Create
+ name: Create
+ elements: [0, 1, 3, 4]
+ - type: PyFilter
+ name: Filter
+ keep: "lambda elem: elem > 2"
+ input: Create
+ output: Filter
+ ''')
+ # No exception raised
+ assert_that(result, equal_to([3, 4]))
+
+ def test_name_is_ambiguous(self):
+ with beam.Pipeline(options=beam.options.pipeline_options.PipelineOptions(
+ pickle_library='cloudpickle')) as p:
+
+ with self.assertRaises(ValueError):
+ result = p | YamlTransform(
+ '''
+ type: composite
+ transforms:
+ - type: Create
+ name: CreateData
+ elements: [0, 1, 3, 4]
+ - type: PyFilter
+ name: PyFilter
+ keep: "lambda elem: elem > 2"
+ input: CreateData
+ - type: PyFilter
+ name: AnotherFilter
+ keep: "lambda elem: elem > 3"
+ input: PyFilter
Review Comment:
I'm fine with this being an error--it's still a bit ambiguous and if we really want to change our minds later we can (but not the other way around).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb merged pull request #26771: Allow for same name and type, if it is not ambiguous in yaml inputs
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb merged PR #26771:
URL: https://github.com/apache/beam/pull/26771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26771: Allow for same name and type, if it is not ambiguous in yaml inputs
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26771:
URL: https://github.com/apache/beam/pull/26771#issuecomment-1553574081
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on pull request #26771: Allow for same name and type, if it is not ambiguous in yaml inputs
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #26771:
URL: https://github.com/apache/beam/pull/26771#issuecomment-1557276808
@robertwb formatting and linter errors are fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org