You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "JoeCMoore (via GitHub)" <gi...@apache.org> on 2023/08/25 14:16:22 UTC
[GitHub] [beam] JoeCMoore commented on a diff in pull request #26889: Update document Storage API Support in `Google BigQuery I/O connector` for Python SDK
JoeCMoore commented on code in PR #26889:
URL: https://github.com/apache/beam/pull/26889#discussion_r1305718914
##########
sdks/python/apache_beam/examples/snippets/snippets.py:
##########
@@ -1181,6 +1189,32 @@ def model_bigqueryio(
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED)
# [END model_bigqueryio_write]
+ # [START model_bigqueryio_write_with_storage_write_api]
+ quotes | beam.io.WriteToBigQuery(
+ table_spec,
+ schema=table_schema,
+ method=beam.io.WriteToBigQuery.Method.STORAGE_WRITE_API)
+ # [END model_bigqueryio_write_with_storage_write_api]
Review Comment:
> I looked into the code again and found [this](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/schemas.py#L787~L799). It seems that `beam:logical_type:millis_instant:v1` is translated as `FieldType.DATETIME` in [SchemaTranslation](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaTranslation.java#L427), which is matched with the [output mapping](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L324). To use `MillisInstant`, `LogicalType.register_logical_type(MillisInstant)` should be added to the pipeline code.
>
> **Example**
>
> ```
> LogicalType.register_logical_type(MillisInstant)
> quotes | beam.io.WriteToBigQuery(
> table_spec,
> schema=table_schema,
> write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE,
> create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
> method=beam.io.WriteToBigQuery.Method.STORAGE_WRITE_API,
> )
> ```
Where does `LogicalType.register_logical_type(MillisInstant)` reference from? I'm trying to find a workaround that's usable for now since this isn't fixed in v2.49.0. My JSON schema is correctly converted to a `bigquery.TableSchema` instance but when passing it to apache beam it gives the error you mentioned above.
In my case the timestamp type is incorrectly translated to datetime (`LOGICAL_TYPE<beam:logical_type:micros_instant:v1>` ➔ `DATETIME NOT NULL`).
I'd love to help solve this, please let me know of anything I can help with for testing etc.
##########
sdks/python/apache_beam/examples/snippets/snippets.py:
##########
@@ -1181,6 +1189,32 @@ def model_bigqueryio(
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED)
# [END model_bigqueryio_write]
+ # [START model_bigqueryio_write_with_storage_write_api]
+ quotes | beam.io.WriteToBigQuery(
+ table_spec,
+ schema=table_schema,
+ method=beam.io.WriteToBigQuery.Method.STORAGE_WRITE_API)
+ # [END model_bigqueryio_write_with_storage_write_api]
Review Comment:
> I looked into the code again and found [this](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/schemas.py#L787~L799). It seems that `beam:logical_type:millis_instant:v1` is translated as `FieldType.DATETIME` in [SchemaTranslation](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaTranslation.java#L427), which is matched with the [output mapping](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L324). To use `MillisInstant`, `LogicalType.register_logical_type(MillisInstant)` should be added to the pipeline code.
>
> **Example**
>
> ```
> LogicalType.register_logical_type(MillisInstant)
> quotes | beam.io.WriteToBigQuery(
> table_spec,
> schema=table_schema,
> write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE,
> create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
> method=beam.io.WriteToBigQuery.Method.STORAGE_WRITE_API,
> )
> ```
Where does `LogicalType.register_logical_type(MillisInstant)` reference from? I'm trying to find a workaround that's usable for now since this isn't fixed in v2.49.0. My JSON schema is correctly converted to a `bigquery.TableSchema` instance but when passing it to apache beam it gives the error you mentioned above.
In my case the timestamp type is incorrectly translated to datetime (`LOGICAL_TYPE<beam:logical_type:micros_instant:v1>` ➔ `DATETIME NOT NULL`).
I'd love to help solve this, please let me know of anything I can help with for testing etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org