You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "liferoad (via GitHub)" <gi...@apache.org> on 2025/06/14 19:30:26 UTC
[PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
liferoad opened a new pull request, #35288:
URL: https://github.com/apache/beam/pull/35288
Fixes #35194
When using ReadFromPubSub with a schema in Beam YAML, the pipeline would fail with a KeyError if a field specified in the schema was missing from the incoming JSON message.
This commit fixes the issue by modifying the `json_to_row` function in `apache_beam/yaml/json_utils.py`. The direct dictionary access `value[name]` is replaced with `value.get(name)` to safely handle missing keys, returning `None` instead of raising an error.
The converters for array, map, and row types have also been made robust to handle `None` values, which can occur for missing optional fields of these complex types.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #35288:
URL: https://github.com/apache/beam/pull/35288#issuecomment-2989138714
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/35288?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 54.51%. Comparing base [(`cecfa61`)](https://app.codecov.io/gh/apache/beam/commit/cecfa6128ceaecf3b8d02a8a42788b3505595845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`c2f2553`)](https://app.codecov.io/gh/apache/beam/commit/c2f2553c1b39d7009b78bfeb58c3b1148cb9c3b2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 39 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #35288 +/- ##
==========================================
Coverage 54.50% 54.51%
Complexity 1559 1559
==========================================
Files 1035 1036 +1
Lines 161595 161782 +187
Branches 1139 1139
==========================================
+ Hits 88084 88189 +105
- Misses 71380 71462 +82
Partials 2131 2131
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/35288/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/apache/beam/pull/35288/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.82% <100.00%> (-0.07%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/35288?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #35288:
URL: https://github.com/apache/beam/pull/35288#discussion_r2151073795
##########
sdks/python/apache_beam/yaml/json_utils.py:
##########
@@ -172,26 +172,34 @@ def json_to_row(beam_type: schema_pb2.FieldType) -> Callable[[Any], Any]:
return lambda value: value
elif type_info == "array_type":
element_converter = json_to_row(beam_type.array_type.element_type)
- return lambda value: [element_converter(e) for e in value]
+ return lambda value: [
+ element_converter(e) for e in value
Review Comment:
Rather than modify each of these cases, I would think that element_converter should accept None for if (and only if) it is a nullable type.
This is probably mostly easily done by putting a case a the top of this function like
```
if beam_type.nullable:
non_null_converer = json_to_row(schema_pb2.FieldType(type_info=beam_type, nullable=False))
return lambda e: None if e is None else non_null_converer(e)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "jonathaningram (via GitHub)" <gi...@apache.org>.
jonathaningram commented on PR #35288:
URL: https://github.com/apache/beam/pull/35288#issuecomment-2978464631
@robertwb is it even possible to define an optional or non-optional field? As described in the original issue #35179, I couldn't work out how to specify "required-ness" on my schema.
I did notice in the tests in this PR that `nullable` was being used and I was going to comment on whether that's something that external pipeline authors are meant to be able to configure, but I removed my comment because I decided that maybe the `nullable` was just to help set up a schema for the tests (and I could prove the fix worked e2e in Dataflow).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #35288:
URL: https://github.com/apache/beam/pull/35288#issuecomment-2976854122
> The code looks right to me and I tested a Dataflow Beam YAML pipeline without this change and with it. I can confirm the key error goes away with this change. I tested a missing Pub/Sub message field but not a missing attribute. I assume it works for both though.
>
> Maybe there could be a corresponding docs update to go with this PR, e.g., tell users what happens if they have missing fields, but leave that with you to decide on.
Good idea. Added this to CHANGES.md.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #35288:
URL: https://github.com/apache/beam/pull/35288#issuecomment-2973044422
@jonathaningram possible to validate this PR from your side? Feel free to review it as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35288:
URL: https://github.com/apache/beam/pull/35288#discussion_r2154879175
##########
sdks/python/apache_beam/yaml/json_utils.py:
##########
@@ -184,14 +191,25 @@ def json_to_row(beam_type: schema_pb2.FieldType) -> Callable[[Any], Any]:
value_converter = json_to_row(beam_type.map_type.value_type)
return lambda value: {k: value_converter(v) for (k, v) in value.items()}
elif type_info == "row_type":
+ fields = {field.name: field for field in beam_type.row_type.schema.fields}
converters = {
- field.name: json_to_row(field.type)
- for field in beam_type.row_type.schema.fields
+ name: json_to_row(field.type)
+ for name, field in fields.items()
}
- return lambda value: beam.Row(
- **
- {name: convert(value[name])
- for (name, convert) in converters.items()})
+
+ def convert_row(value):
+ kwargs = {}
+ for name, convert in converters.items():
+ field = fields[name]
Review Comment:
The test is failing because we are trying to pickle the actual message descriptor in the closure.
I tested this which works.
```
elif type_info == "row_type":
field_nullable_status = {
field.name: field.type.nullable
for field in beam_type.row_type.schema.fields
}
converters = {
field.name: json_to_row(field.type)
for field in beam_type.row_type.schema.fields
}
def convert_row(value):
kwargs = {}
for name, convert in converters.items():
if name in value:
kwargs[name] = convert(value[name])
elif field_nullable_status[name]:
kwargs[name] = convert(None)
else:
raise KeyError(f"Missing required field: {name}")
return beam.Row(**kwargs)
return convert_row
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #35288:
URL: https://github.com/apache/beam/pull/35288#issuecomment-2978490400
By default, all properties in a json schema are optional; to declare them otherwise one uses the required field: https://json-schema.org/understanding-json-schema/reference/object#required which we respect in Beam: https://github.com/apache/beam/blob/release-2.65/sdks/python/apache_beam/yaml/json_utils.py#L67 .
This function takes as input a schema_pb2.FieldType and should respect whether the types in question are optional (though I'm not saying it might not be to strict now).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #35288:
URL: https://github.com/apache/beam/pull/35288#issuecomment-2978494821
> Drive by comment: it'd be nice if there was a test ensuring we still fail for _non_ optional fields.
Good point. The original PR indeed did not force this requirement. I updated the code to check the required fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35288:
URL: https://github.com/apache/beam/pull/35288#issuecomment-2973230257
Assigning reviewers:
R: @claudevdm for label python.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #35288:
URL: https://github.com/apache/beam/pull/35288#discussion_r2151073795
##########
sdks/python/apache_beam/yaml/json_utils.py:
##########
@@ -172,26 +172,34 @@ def json_to_row(beam_type: schema_pb2.FieldType) -> Callable[[Any], Any]:
return lambda value: value
elif type_info == "array_type":
element_converter = json_to_row(beam_type.array_type.element_type)
- return lambda value: [element_converter(e) for e in value]
+ return lambda value: [
+ element_converter(e) for e in value
Review Comment:
Rather than modify each of these cases, I would think that element_converter should accept None for nullable types (and reject it otherwise).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad merged PR #35288:
URL: https://github.com/apache/beam/pull/35288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix(yaml): Handle missing optional fields in JSON parsing [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on code in PR #35288:
URL: https://github.com/apache/beam/pull/35288#discussion_r2157607229
##########
sdks/python/apache_beam/yaml/json_utils.py:
##########
@@ -184,14 +191,25 @@ def json_to_row(beam_type: schema_pb2.FieldType) -> Callable[[Any], Any]:
value_converter = json_to_row(beam_type.map_type.value_type)
return lambda value: {k: value_converter(v) for (k, v) in value.items()}
elif type_info == "row_type":
+ fields = {field.name: field for field in beam_type.row_type.schema.fields}
converters = {
- field.name: json_to_row(field.type)
- for field in beam_type.row_type.schema.fields
+ name: json_to_row(field.type)
+ for name, field in fields.items()
}
- return lambda value: beam.Row(
- **
- {name: convert(value[name])
- for (name, convert) in converters.items()})
+
+ def convert_row(value):
+ kwargs = {}
+ for name, convert in converters.items():
+ field = fields[name]
Review Comment:
Thanks a lot! Good catch! Applied your changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org