You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "robertwb (via GitHub)" <gi...@apache.org> on 2023/09/02 00:34:02 UTC
[GitHub] [beam] robertwb opened a new pull request, #28289: Better errors when inputs are omitted.
robertwb opened a new pull request, #28289:
URL: https://github.com/apache/beam/pull/28289
It's not always possible to know if a transform consumes inputs, or can act as a root transform (and in fact some may be able to do both depending on their configuration), but when a transform expecting inputs doesn't get them the error can be quite obscure. This adds best-effort checking and a better error in that case.
We also allow explicitly setting empty imputs to work around this error (which is where most of the complexity of this change lies). Importantly, sources (no matter their name) are not required to have inputs.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on a diff in pull request #28289: Better errors when inputs are omitted.
Posted by "byronellis (via GitHub)" <gi...@apache.org>.
byronellis commented on code in PR #28289:
URL: https://github.com/apache/beam/pull/28289#discussion_r1328944486
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -61,6 +61,16 @@ def provided_transforms(self) -> Iterable[str]:
"""Returns a list of transform type names this provider can handle."""
raise NotImplementedError(type(self))
+ def requires_inputs(self, typ: str, args: Mapping[str, Any]) -> bool:
+ """Returns whether this transform requires inputs.
+
+ Specifically, if this returns True and inputs are not provided than an error
+ will be thrown.
+
+ This is best-effort, primarily for better and earlier error messages.
+ """
+ return not typ.startswith('Read')
Review Comment:
Presuming this is to provide a reasonable default? Seems like it might lead to surprising false positives though?
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -138,8 +146,19 @@ def create_transform(self, type, args, yaml_create_transform):
except Exception:
# It's possible this service doesn't vend schema transforms.
self._schema_transforms = {}
+ return self._schema_transforms
+
+ def requires_inputs(self, typ, args):
+ if self._urns[type] in self.schema_transforms():
+ return bool(self.schema_transforms()[self._urns[type]].inputs)
Review Comment:
Unfortunately relying on schema transform's inputs/outputs is not very reliable (and maybe completely unreliable). They do not depend on the configuration passed to the transforms making them basically useless for determining anything other than inputs/outputs that are present in all possible configurations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #28289: Better errors when inputs are omitted.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #28289:
URL: https://github.com/apache/beam/pull/28289#discussion_r1329148697
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -138,8 +146,19 @@ def create_transform(self, type, args, yaml_create_transform):
except Exception:
# It's possible this service doesn't vend schema transforms.
self._schema_transforms = {}
+ return self._schema_transforms
+
+ def requires_inputs(self, typ, args):
+ if self._urns[type] in self.schema_transforms():
+ return bool(self.schema_transforms()[self._urns[type]].inputs)
Review Comment:
Yeah, it'd be nice to be able to query about the requirements of a configured transform before invoking it, but that's a larger change... Most (though not all) transforms have fixed inputs at least, so this'll cover those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #28289: Better errors when inputs are omitted.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28289:
URL: https://github.com/apache/beam/pull/28289#issuecomment-1703581233
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #28289: Better errors when inputs are omitted.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #28289:
URL: https://github.com/apache/beam/pull/28289#issuecomment-1703578076
R: @byronellis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #28289: Better errors when inputs are omitted.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #28289:
URL: https://github.com/apache/beam/pull/28289#discussion_r1329144050
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -61,6 +61,16 @@ def provided_transforms(self) -> Iterable[str]:
"""Returns a list of transform type names this provider can handle."""
raise NotImplementedError(type(self))
+ def requires_inputs(self, typ: str, args: Mapping[str, Any]) -> bool:
+ """Returns whether this transform requires inputs.
+
+ Specifically, if this returns True and inputs are not provided than an error
+ will be thrown.
+
+ This is best-effort, primarily for better and earlier error messages.
+ """
+ return not typ.startswith('Read')
Review Comment:
Yep, just a reasonable default. Most transforms that do not start with Read require an input by, the main exception being SQL (where we can't tell and the "typical" use of selecting on PCollection would be good to error on). One can explicitly state `inputs: []` to suppress this error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb merged pull request #28289: Better errors when inputs are omitted.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb merged PR #28289:
URL: https://github.com/apache/beam/pull/28289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #28289: Better errors when inputs are omitted.
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #28289:
URL: https://github.com/apache/beam/pull/28289#issuecomment-1703607602
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28289?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#28289](https://app.codecov.io/gh/apache/beam/pull/28289?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (604987d) into [master](https://app.codecov.io/gh/apache/beam/commit/f91bb68b2e9df3788914898c03fdf42445f912d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (f91bb68) will **increase** coverage by `0.08%`.
> Report is 39 commits behind head on master.
> The diff coverage is `81.13%`.
```diff
@@ Coverage Diff @@
## master #28289 +/- ##
==========================================
+ Coverage 72.29% 72.37% +0.08%
==========================================
Files 678 679 +1
Lines 99911 100218 +307
==========================================
+ Hits 72233 72536 +303
- Misses 26112 26116 +4
Partials 1566 1566
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/28289/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/apache/beam/pull/28289/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.92% <81.13%> (+0.07%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/beam/pull/28289?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/yaml/yaml\_provider.py](https://app.codecov.io/gh/apache/beam/pull/28289?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0veWFtbC95YW1sX3Byb3ZpZGVyLnB5) | `70.13% <61.90%> (+0.16%)` | :arrow_up: |
| [sdks/python/apache\_beam/yaml/yaml\_transform.py](https://app.codecov.io/gh/apache/beam/pull/28289?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0veWFtbC95YW1sX3RyYW5zZm9ybS5weQ==) | `90.23% <93.75%> (+0.30%)` | :arrow_up: |
... and [15 files with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/28289/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org