You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 21:53:28 UTC
[GitHub] [beam] damccorm opened a new issue, #21208: BigQueryUtils doesn't process List or Map with nullables
damccorm opened a new issue, #21208:
URL: https://github.com/apache/beam/issues/21208
When using PubsubAvroToBigQuery Dataflow template, [I noticed this issue. ](https://github.com/GoogleCloudPlatform/DataflowTemplates/issues/287)
This seems to be related to the way [toBeamRow](https://github.com/apache/beam/blob/v2.32.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L593) is handling TableRow with List containing null values.
The error trace has this path (I only added the relevant ones):
a. com.google.cloud.teleport.v2.transforms.BigQueryConverters$TableRowToGenericRecordFn.apply(BigQueryConverters.java:548)
b. [org.apache.beam.sdk.io](https://org.apache.beam.sdk.io/).gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:580)
c. [org.apache.beam.sdk.io](https://org.apache.beam.sdk.io/).gcp.bigquery.BigQueryUtils.toBeamRowFieldValue(BigQueryUtils.java:593)
d. [org.apache.beam.sdk.io](https://org.apache.beam.sdk.io/).gcp.bigquery.BigQueryUtils.toBeamValue(BigQueryUtils.java:641)
On c., a validation of null "bqValue" objects is performed; however, it appears that some of the elements are of List type (for BigQuery Record type); therefore, the List object is validated to be non-null, but not the elements of the List.
On d., the method is executed recursively to process all the elements of the List object; however, it seems that some objects are Null so this method is throwing the NullPointerExceptions.
*toBeamValue method should probably not used toBeamValue recursively but toBeamRowFieldValue instead.*
Imported from Jira [BEAM-12863](https://issues.apache.org/jira/browse/BEAM-12863). Original Jira may contain additional context.
Reported by: pocman.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #21208: BigQueryUtils doesn't process List or Map with nullables
Posted by GitBox <gi...@apache.org>.
damccorm commented on issue #21208:
URL: https://github.com/apache/beam/issues/21208#issuecomment-1228727017
Hey @iamdebojyoti as far as I know this is still an issue, and we'd be happy to have you contribute. With that said, I just moved the issue [from Jira](https://issues.apache.org/jira/browse/BEAM-12863) to Issues I haven't actually run into it myself, I believe the original issue was created by @ecapoccia
Since its somewhat old, I'd start y seeing if you can repro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iamdebojyoti commented on issue #21208: BigQueryUtils doesn't process List or Map with nullables
Posted by GitBox <gi...@apache.org>.
iamdebojyoti commented on issue #21208:
URL: https://github.com/apache/beam/issues/21208#issuecomment-1228703194
Hi @damccorm
Is this issue still open? If yes, I would like to contribute to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jon-esperanza commented on issue #21208: BigQueryUtils doesn't process List or Map with nullables
Posted by "jon-esperanza (via GitHub)" <gi...@apache.org>.
jon-esperanza commented on issue #21208:
URL: https://github.com/apache/beam/issues/21208#issuecomment-1646725107
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn closed issue #21208: BigQueryUtils doesn't process List or Map with nullables
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #21208: BigQueryUtils doesn't process List or Map with nullables
URL: https://github.com/apache/beam/issues/21208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jon-esperanza commented on issue #21208: BigQueryUtils doesn't process List or Map with nullables
Posted by "jon-esperanza (via GitHub)" <gi...@apache.org>.
jon-esperanza commented on issue #21208:
URL: https://github.com/apache/beam/issues/21208#issuecomment-1664769147
@Abacn can this issue be closed as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org