You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "riteshghorse (via GitHub)" <gi...@apache.org> on 2023/10/11 17:41:17 UTC
[PR] fail windowInto when there are no inputs to it [beam]
riteshghorse opened a new pull request, #28946:
URL: https://github.com/apache/beam/pull/28946
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Fail WindowInto when there are no inputs to it [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28946:
URL: https://github.com/apache/beam/pull/28946#issuecomment-1759603161
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fail windowInto when there are no inputs to it [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28946:
URL: https://github.com/apache/beam/pull/28946#issuecomment-1759600557
R: @tvalentyn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Fail WindowInto when there are no inputs to it [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb merged PR #28946:
URL: https://github.com/apache/beam/pull/28946
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Fail WindowInto when there are no inputs to it [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #28946:
URL: https://github.com/apache/beam/pull/28946#discussion_r1362363985
##########
sdks/python/apache_beam/runners/common.py:
##########
@@ -1928,6 +1928,10 @@ def validate_transform(transform_id):
"Incompatible input coder %s and output coder %s for transform %s" %
(transform_id, input_coder, output_coder))
+ if transform_proto.spec.urn == common_urns.primitives.ASSIGN_WINDOWS.urn:
Review Comment:
Let's validate `common_urns.primitives.PAR_DO.urn` as well.
##########
sdks/python/apache_beam/runners/common.py:
##########
@@ -1928,6 +1928,10 @@ def validate_transform(transform_id):
"Incompatible input coder %s and output coder %s for transform %s" %
(transform_id, input_coder, output_coder))
+ if transform_proto.spec.urn == common_urns.primitives.ASSIGN_WINDOWS.urn:
+ if not transform_proto.inputs:
+ raise ValueError("Unexpected number of inputs: %s" % transform_proto)
Review Comment:
Maybe say "Missing inputs?"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Fail WindowInto when there are no inputs to it [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28946:
URL: https://github.com/apache/beam/pull/28946#discussion_r1357225166
##########
sdks/python/apache_beam/pipeline.py:
##########
@@ -1353,18 +1353,28 @@ def transform_to_runner_api(
environment_id = context.get_environment_id_for_resource_hints(
self.resource_hints)
+ inputs = {
Review Comment:
> p | beam.WindowInto(...) should also be an error right there (checked in WindowInto's expand method).
+1 to Robert's comment, moving this validation inside `WindowInto` methods would be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fail windowInto when there are no inputs to it [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #28946:
URL: https://github.com/apache/beam/pull/28946#issuecomment-1758205365
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28946?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#28946](https://app.codecov.io/gh/apache/beam/pull/28946?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (eb13b0b) into [master](https://app.codecov.io/gh/apache/beam/commit/2bfcb9fda37be8d5b9550a1b118ed87d03fdf12c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (2bfcb9f) will **decrease** coverage by `0.02%`.
> Report is 1 commits behind head on master.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## master #28946 +/- ##
==========================================
- Coverage 72.16% 72.15% -0.02%
==========================================
Files 686 686
Lines 101560 101593 +33
==========================================
+ Hits 73294 73302 +8
- Misses 26687 26712 +25
Partials 1579 1579
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/28946/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/apache/beam/pull/28946/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.58% <100.00%> (-0.03%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files](https://app.codecov.io/gh/apache/beam/pull/28946?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/pipeline.py](https://app.codecov.io/gh/apache/beam/pull/28946?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `91.78% <100.00%> (+0.03%)` | :arrow_up: |
... and [7 files with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/28946/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fail windowInto when there are no inputs to it [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28946:
URL: https://github.com/apache/beam/pull/28946#issuecomment-1758276923
Run Python 3.8 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Fail WindowInto when there are no inputs to it [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #28946:
URL: https://github.com/apache/beam/pull/28946#issuecomment-1759926310
FWIW, I think this should be fixed in the service rather than adding a special case to the SDK (Python only, which won't be out for 6 weeks or so).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Fail WindowInto when there are no inputs to it [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #28946:
URL: https://github.com/apache/beam/pull/28946#issuecomment-1760126482
A good place to validate this might be https://github.com/apache/beam/blob/release-2.51.0/sdks/python/apache_beam/runners/common.py#L1895
We could have a mapping of known URNs -> expected input/output cardinality (if any), and flag violations. (We could ensure the inputs/outputs actually exist.)
`p | beam.WindowInto(...)` should also be an error right there (checked in WindowInto's expand method).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org