You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2022/05/30 17:00:00 UTC
[jira] [Updated] (BEAM-13990) BigQueryIO cannot write to DATE and TIMESTAMP columns when using Storage Write API
[ https://issues.apache.org/jira/browse/BEAM-13990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Beam JIRA Bot updated BEAM-13990:
---------------------------------
Labels: stale-assigned (was: )
> BigQueryIO cannot write to DATE and TIMESTAMP columns when using Storage Write API
> -----------------------------------------------------------------------------------
>
> Key: BEAM-13990
> URL: https://issues.apache.org/jira/browse/BEAM-13990
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Affects Versions: 2.36.0
> Reporter: Du Liu
> Assignee: Du Liu
> Priority: P2
> Labels: stale-assigned
> Original Estimate: 120h
> Time Spent: 20h 40m
> Remaining Estimate: 99h 20m
>
> when using Storage Write API with BigQueryIO, DATE and TIMESTAMP values are currently converted to String type in protobuf message. This is incorrect, according to storage write api [documentation|#data_type_conversions],] DATE should be converted to int32 and TIMESTAMP should be converted to int64.
> Here's error message:
> INFO: Stream finished with error com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The proto field mismatched with BigQuery field at D6cbe536b_4dab_4292_8fda_ff2932dded49.datevalue, the proto field type string, BigQuery field type DATE Entity
> I have included an integration test here: [https://github.com/liu-du/beam/commit/b56823d1d213adf6ca5564ce1d244cc4ae8f0816]
>
> The problem is because DATE and TIMESTAMP are converted to String in protobuf message here:
> [https://github.com/apache/beam/blob/a78fec72d0d9198eef75144a7bdaf93ada5abf9b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L69]
>
> Storage Write API reject the request because it's expecting int32/int64 values.
>
> I've opened a PR here: https://github.com/apache/beam/pull/16926
--
This message was sent by Atlassian Jira
(v8.20.7#820007)