You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/21 15:24:33 UTC
[GitHub] [beam] esciara opened a new issue, #21961: [Feature Request]: Add support for BigQuery Storge Write API in Python
esciara opened a new issue, #21961:
URL: https://github.com/apache/beam/issues/21961
### What would you like to happen?
Much like #20687 to read to BigQuery, The Java SDK contains an implementation which uses the BigQuery Storage API to write to BigQuery. We should implement the same for Python.
In our case, we would like to use it instead of using the `FILE_LOADS` ingestion method in streaming mode, where errors during the loading process are not propagated to the pipeline. We are currently using it as we do not need `STREAMING_INSERT` features, and hence want to save on cost.
### Issue Priority
Priority: 3
### Issue Component
Component: io-py-gcp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #21961: [Feature Request]: Add support for BigQuery Storge Write API in Python
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #21961:
URL: https://github.com/apache/beam/issues/21961#issuecomment-1335861812
@chamikaramj FYI for if/when this can be done via xlang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rizenfrmtheashes commented on issue #21961: [Feature Request]: Add support for BigQuery Storge Write API in Python
Posted by GitBox <gi...@apache.org>.
rizenfrmtheashes commented on issue #21961:
URL: https://github.com/apache/beam/issues/21961#issuecomment-1258673784
I took a look at this, and it seems like the bigquery storage api has a straightforward [jsonstreamwriter but for java](https://cloud.google.com/bigquery/docs/write-api#java_client). [for python, it only exists as protobufs](https://cloud.google.com/bigquery/docs/write-api#python_client). Hence why the bigquery storage write api for dataflow in python is missing, but exists in java. That api will likely need to be made before the dataflow functions can be made by the core contributors group. Sad that [this blog post made it seem like it was in GA for everything](https://cloud.google.com/blog/products/data-analytics/streaming-data-into-bigquery-using-storage-write-api).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj closed issue #21961: [Feature Request]: Add support for BigQuery Storge Write API in Python
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj closed issue #21961: [Feature Request]: Add support for BigQuery Storge Write API in Python
URL: https://github.com/apache/beam/issues/21961
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org