You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 23:33:20 UTC
[GitHub] [beam] damccorm opened a new issue, #21456: BigQueryIO cannot write to DATE and TIMESTAMP columns when using Storage Write API
damccorm opened a new issue, #21456:
URL: https://github.com/apache/beam/issues/21456
when using Storage Write API with BigQueryIO, DATE and TIMESTAMP values are currently converted to String type in protobuf message. This is incorrect, according to storage write api [documentation|#data_type_conversions],] DATE should be converted to int32 and TIMESTAMP should be converted to int64.
Here's error message:
INFO: Stream finished with error com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The proto field mismatched with BigQuery field at D6cbe536b_4dab_4292_8fda_ff2932dded49.datevalue, the proto field type string, BigQuery field type DATE Entity
I have included an integration test here: [https://github.com/liu-du/beam/commit/b56823d1d213adf6ca5564ce1d244cc4ae8f0816](https://github.com/liu-du/beam/commit/b56823d1d213adf6ca5564ce1d244cc4ae8f0816)
The problem is because DATE and TIMESTAMP are converted to String in protobuf message here:
[https://github.com/apache/beam/blob/a78fec72d0d9198eef75144a7bdaf93ada5abf9b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L69](https://github.com/apache/beam/blob/a78fec72d0d9198eef75144a7bdaf93ada5abf9b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L69)
Storage Write API reject the request because it's expecting int32/int64 values.
I've opened a PR here: https://github.com/apache/beam/pull/16926
Imported from Jira [BEAM-13990](https://issues.apache.org/jira/browse/BEAM-13990). Original Jira may contain additional context.
Reported by: duliu.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] closed issue #21456: BigQueryIO cannot write to DATE and TIMESTAMP columns when using Storage Write API
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #21456: BigQueryIO cannot write to DATE and TIMESTAMP columns when using Storage Write API
URL: https://github.com/apache/beam/issues/21456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on issue #21456: BigQueryIO cannot write to DATE and TIMESTAMP columns when using Storage Write API
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on issue #21456:
URL: https://github.com/apache/beam/issues/21456#issuecomment-1310957146
.close-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #21456: BigQueryIO cannot write to DATE and TIMESTAMP columns when using Storage Write API
Posted by GitBox <gi...@apache.org>.
damccorm commented on issue #21456:
URL: https://github.com/apache/beam/issues/21456#issuecomment-1146702838
Unable to assign user @liu-du. If able, self-assign, otherwise tag @damccorm so that he can assign you. Because of GitHub's spam prevention system, your activity is required to enable assignment in this repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on issue #21456: BigQueryIO cannot write to DATE and TIMESTAMP columns when using Storage Write API
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on issue #21456:
URL: https://github.com/apache/beam/issues/21456#issuecomment-1310957098
This was fixed in #17404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org