You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/03 17:44:15 UTC

[GitHub] [beam] takaaki7 opened a new issue, #23025: [Bug]: Cannot use Json type with BigQueryIO and write method FILE_LOAD, avro

takaaki7 opened a new issue, #23025:
URL: https://github.com/apache/beam/issues/23025

   ### What happened?
   
   BigQueryIO document says FILE_LOAD not support JSON type.
   https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1841
   
   And when specifying schema to use avro, validation logic causes error.
   https://github.com/apache/beam/blob/948af30a5b665fe74b7052b673e95ff5f5fc426a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L3078
   
   But currently, It seems JSON type works well with load api.
   
   I think now we can remove this validation and modify document.
   
   ### Issue Priority
   
   Priority: 3
   
   ### Issue Component
   
   Component: extensions-java-gcp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Cannot use Json type with BigQueryIO and write method FILE_LOAD, avro [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #23025:
URL: https://github.com/apache/beam/issues/23025#issuecomment-1877787215

   Checked that this feature was added in late 2022 (internal tracker cl/457809718). So it is safe to lift this checkArgument. Opened #29923


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Cannot use Json type with BigQueryIO and write method FILE_LOAD, avro [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #23025: [Bug]: Cannot use Json type with BigQueryIO and write method FILE_LOAD, avro
URL: https://github.com/apache/beam/issues/23025


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Cannot use Json type with BigQueryIO and write method FILE_LOAD, avro [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #23025:
URL: https://github.com/apache/beam/issues/23025#issuecomment-1877762443

   I tried to remove the validation and use run BigQueryIOJsonIT. There is a caveat that for a TableRow, if I set it to be
   ```
   .set("country_code", some_json_of_string_type) // some_json_of_string_type = "{\"a\":\"b\"}"
   ```
   The written json is a single escaped string ( "{\"a\":\"b\"}" ) using FILE_LOAD method. But the string is unescaped using STORAGE_WRITE_API or STREAMING_INSERT method ( {"a":"b"} ).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org