You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/11/09 16:54:34 UTC

[GitHub] [beam] carlpayne opened a new issue, #24063: [Feature Request]: Support SchemaUpdateOptions for BigQueryIO Storage Write API

carlpayne opened a new issue, #24063:
URL: https://github.com/apache/beam/issues/24063

   ### What would you like to happen?
   
   When writing data with BigQueryIO using `BigQueryIO.Write.Method.STORAGE_API_AT_LEAST_ONCE` or `BigQueryIO.Write.Method.STORAGE_WRITE_API`, it would be very useful to be able to specify one or more `SchemaUpdateOption` values via `.withSchemaUpdateOptions`. 
   
   Currently this only works for `BigQueryIO.Write.Method.FILE_LOADS`, which means that for other insert methods new fields are either ignored (if `ignoreUnknownValues` is set), or fail to insert. In our use case, we are streaming Avro records from Kafka to BigQuery, with the schema constantly changing (mostly new fields). Having BigQueryIO manage this would be a huge benefit.
   
   ### Issue Priority
   
   Priority: 3
   
   ### Issue Component
   
   Component: io-java-gcp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] kennknowles commented on issue #24063: [Feature Request]: Support SchemaUpdateOptions for BigQueryIO Storage Write API

Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #24063:
URL: https://github.com/apache/beam/issues/24063#issuecomment-1314364960

   CC @reuvenlax @scwhittle 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] kennknowles commented on issue #24063: [Feature Request]: Support SchemaUpdateOptions for BigQueryIO Storage Write API

Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #24063:
URL: https://github.com/apache/beam/issues/24063#issuecomment-1314365199

   @johnjcasey 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org