[jira] [Commented] (BEAM-11648) Implement new BigQuery sink (Vortex)

["Marcin Kuthan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-11648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441559#comment-17441559 ] 

Marcin Kuthan commented on BEAM-11648:
--------------------------------------

Existing Streaming API is marked as deprecated: [https://cloud.google.com/bigquery/streaming-data-into-bigquery.] BigQuery documentation recommends new Storage Write API but Beam seems to be far, far behind. It looks that implementation already exists: [https://beam.apache.org/releases/javadoc/2.33.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.Method.html#STORAGE_WRITE_API] but:
 # Documentation is missing (JavaDoc "Use the new, experimental Storage Write API." is not enough, really).
 # I don't know which BigQueryIO features are supported in STORAGE_WRITE_API (e.g. dynamic destination, retry policies, insert ids, invalid rows handling, beam schema, auto sharding, clustering, partitioning, etc.).
 # I don't know which Beam version supports new API (Jira "fix version" is not set)

Existing Streaming Insert method has major limitations (low throughput, high costs), we really need decent support for Storage Write API.

> Implement new BigQuery sink (Vortex)
> ------------------------------------
>
>                 Key: BEAM-11648
>                 URL: https://issues.apache.org/jira/browse/BEAM-11648
>             Project: Beam
>          Issue Type: New Feature
>          Components: extensions-java-gcp
>            Reporter: Reuven Lax
>            Priority: P3
>          Time Spent: 49h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)