You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "lucasnoetzold (via GitHub)" <gi...@apache.org> on 2024/04/04 15:05:02 UTC
[I] [Bug]: ClassCastException with unknown fields in STORAGE_WRITE_API [beam]
lucasnoetzold opened a new issue, #30850:
URL: https://github.com/apache/beam/issues/30850
### What happened?
We're gettig a `ClassCastException` when trying to write data to a BigQuery table with the STORAGE_WRITE_API. This only happens with unknown fields on the payload, both `ignoreUnknownValues` and `withAutoSchemaUpdate` options are enabled and the field being written is a REPEATED RECORD.
**Conditions:**
- BigQuery table that contains a "RECORD" type field
- the record field must be on mode "repeated"
- `.ignoreUnknownValues()` option enabled
- `.withAutoSchemaUpdate(true)` option enabled
- payload that contains any unknown field inside the RECORD field that does exist
**Expected Behavior:** unknown fields are ignored and the known payload is sent to BigQuery.
---
I've managed to reproduce the error with this minimal code:
```java
var nested = List.of(Map.of("nested_field_that_exists", "something"));
var row = new TableRow().set("root_field", nested);
var pipeline = Pipeline.create();
pipeline.apply(Create.<TableRow>of(row))
.apply(BigQueryIO.<TableRow>write()
.ignoreUnknownValues()
.withAutoSchemaUpdate(true)
.withFormatFunction(identity())
.withMethod(BigQueryIO.Write.Method.STORAGE_WRITE_API)
.to(new TableReference()
.setProjectId("the_project_id")
.setDatasetId("the_dataset_id")
.setTableId("test_table")));
pipeline.run();
```
The BigQuery table schema used is the following:
Looking at the stack we can see that it happens when it's handling the unknown fields. Since the unknown field is nested, and the field descriptor is inherited from the root field, it expects it to be a list (since the root field is at mode REPEATED):
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ClassCastException with REPEATED RECORD and STORAGE_WRITE_API [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 closed issue #30850: [Bug]: ClassCastException with REPEATED RECORD and STORAGE_WRITE_API
URL: https://github.com/apache/beam/issues/30850
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ClassCastException with REPEATED RECORD and STORAGE_WRITE_API [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #30850:
URL: https://github.com/apache/beam/issues/30850#issuecomment-2138239152
This happens because `TableRowToStorageApiProto` always marks a STRUCT field as an unknown and gives it a value of an empty TableRow (later to be populated). When there are actually no unknown nested fields, we never clean up this empty TableRow (more details in #31447). Some weird conversion happens down the line that leads to the error you're seeing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org