You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "robertwb (via GitHub)" <gi...@apache.org> on 2024/11/12 22:55:03 UTC
[PR] [YAML] Add a new StripErrorMetadata transform. [beam]
robertwb opened a new pull request, #33094:
URL: https://github.com/apache/beam/pull/33094
Beam Yaml's error handling framework returns per-record errors as a schema'd PCollection with associated error metadata (e.g. error messages, tracebacks). Currently there is no way to "unnest" the nested rececords (except for field by field) back to the top level if one wants to re-process these records (or otherwise ignore the metadata). Even if there was a way to do this "up-one-level" unnesting it's not clear that this would be obvious to users to find. Worse, various forms of error handling are not consistent in what the "bad records" schema is, or even where the original record is found (though we do have a caveat in the docs that this is still not set in stone).
This adds a simple, easy to identify transform that abstracts all of these complexities away for the basic usecase.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] Add a new StripErrorMetadata transform. [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #33094:
URL: https://github.com/apache/beam/pull/33094#discussion_r1840909138
##########
sdks/python/apache_beam/yaml/yaml_mapping.py:
##########
@@ -482,6 +483,65 @@ def expand(pcoll, error_handling=None, **kwargs):
return expand
+class _StripErrorMetadata(beam.PTransform):
+ """Strips error metadata from outputs returned via error handling.
+
+ Generally the error outputs for transformations return information about
+ the error encountered (e.g. error messages and tracebacks) in addition to the
+ failing element itself. This transformation attempts to remove that metadata
+ and returns the bad element alone which can be useful for re-processing.
+
+ For example, in the following pipeline snippet::
+
+ - name: MyMappingTransform
+ type: MapToFields
+ input: SomeInput
+ config:
+ language: python
+ fields:
+ ...
+ error_handling:
+ output: errors
+
+ - name: RecoverOriginalElements
+ type: StripErrorMetadata
+ input: MyMappingTransform.errors
+
+ the output of `RecoverOriginalElements` will contain exactly those elements
+ from SomeInput that failed to processes (whereas `MyMappingTransform.errors`
+ would contain those elements paired with error information).
+
+ Note that this relies on the preceding transform actually returning the
+ failing input in a schema'd way. Most built-in transformation follow the
+ correct conventions.
+ """
+
+ _ERROR_FIELD_NAMES = ('failed_row', 'element', 'record')
+
+ def expand(self, pcoll):
+ try:
+ existing_fields = {
+ fld.name: fld.type
+ for fld in schema_from_element_type(pcoll.element_type).fields
+ }
+ except TypeError:
+ fld = None
+ else:
+ for fld in self._ERROR_FIELD_NAMES:
+ if fld in existing_fields:
+ break
+ else:
+ raise ValueError(
+ f"No field name matches one of {self._ERROR_FIELD_NAMES}")
Review Comment:
I think we should expand on this error message and explain what it means (something like `The previous transform does not support the `StripErrorMetadata` transform, and must output error records in XYZ format`).
I'm not going to block submission of the PR on this, since I think it is valuable to make the cut.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] Add a new StripErrorMetadata transform. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33094:
URL: https://github.com/apache/beam/pull/33094#issuecomment-2471785663
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] Add a new StripErrorMetadata transform. [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #33094:
URL: https://github.com/apache/beam/pull/33094#discussion_r1841357311
##########
sdks/python/apache_beam/yaml/yaml_mapping.py:
##########
@@ -482,6 +483,65 @@ def expand(pcoll, error_handling=None, **kwargs):
return expand
+class _StripErrorMetadata(beam.PTransform):
+ """Strips error metadata from outputs returned via error handling.
+
+ Generally the error outputs for transformations return information about
+ the error encountered (e.g. error messages and tracebacks) in addition to the
+ failing element itself. This transformation attempts to remove that metadata
+ and returns the bad element alone which can be useful for re-processing.
+
+ For example, in the following pipeline snippet::
+
+ - name: MyMappingTransform
+ type: MapToFields
+ input: SomeInput
+ config:
+ language: python
+ fields:
+ ...
+ error_handling:
+ output: errors
+
+ - name: RecoverOriginalElements
+ type: StripErrorMetadata
+ input: MyMappingTransform.errors
+
+ the output of `RecoverOriginalElements` will contain exactly those elements
+ from SomeInput that failed to processes (whereas `MyMappingTransform.errors`
+ would contain those elements paired with error information).
+
+ Note that this relies on the preceding transform actually returning the
+ failing input in a schema'd way. Most built-in transformation follow the
+ correct conventions.
+ """
+
+ _ERROR_FIELD_NAMES = ('failed_row', 'element', 'record')
+
+ def expand(self, pcoll):
+ try:
+ existing_fields = {
+ fld.name: fld.type
+ for fld in schema_from_element_type(pcoll.element_type).fields
+ }
+ except TypeError:
+ fld = None
+ else:
+ for fld in self._ERROR_FIELD_NAMES:
+ if fld in existing_fields:
+ break
+ else:
+ raise ValueError(
+ f"No field name matches one of {self._ERROR_FIELD_NAMES}")
Review Comment:
Sure. https://github.com/apache/beam/pull/33110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] Add a new StripErrorMetadata transform. [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #33094:
URL: https://github.com/apache/beam/pull/33094#issuecomment-2471781535
R: @Polber
CC: @damccorm
It'd be nice to get this in the release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [YAML] Add a new StripErrorMetadata transform. [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #33094:
URL: https://github.com/apache/beam/pull/33094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org