You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Luke Cwik (JIRA)" <ji...@apache.org> on 2018/08/08 23:56:00 UTC
[jira] [Resolved] (BEAM-4835) Add more flexible options for data
loading to BigQueryIO.Write
[ https://issues.apache.org/jira/browse/BEAM-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Cwik resolved BEAM-4835.
-----------------------------
Resolution: Fixed
Fix Version/s: 2.7.0
> Add more flexible options for data loading to BigQueryIO.Write
> --------------------------------------------------------------
>
> Key: BEAM-4835
> URL: https://issues.apache.org/jira/browse/BEAM-4835
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Gene Peters
> Assignee: Gene Peters
> Priority: Minor
> Labels: pull-request-available
> Fix For: 2.7.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> As part of the BigQuery API, there are a few options exposed to end-users which allow for more flexible data loading.
> For both [streaming|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setIgnoreUnknownValues-java.lang.Boolean-] and [batch|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/JobConfigurationLoad.html#setIgnoreUnknownValues-java.lang.Boolean-] inserts, the flag "ignoreUnknownValues" can be set, which indicates if BigQuery should accept rows that contain values that do not match the schema.
> [In addition,|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setSkipInvalidRows-java.lang.Boolean-] streaming inserts allow for the option of accepting an inserted batch of rows even if some of of the rows are invalid.
> I've made the necessary code changes to make this available within BigQueryIO.Write and will be attaching the pull request to this ticket for review. Both flags are off by default.
> Let me know if you have any questions or feedback about this!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)