[jira] [Created] (BEAM-10826) Expose BigQuery schema autodetect in Java SDK

["Marcus Truscello (Jira)" <ji...@apache.org>]

Marcus Truscello created BEAM-10826:
---------------------------------------

             Summary: Expose BigQuery schema autodetect in Java SDK
                 Key: BEAM-10826
                 URL: https://issues.apache.org/jira/browse/BEAM-10826
             Project: Beam
          Issue Type: Improvement
          Components: sdk-java-core
            Reporter: Marcus Truscello


The Beam Java SDK's BigQueryIO transform currently doesn't expose the [schema autodetect job configuration|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/JobConfigurationLoad.html#setAutodetect-java.lang.Boolean-].  The feature is exposed by the current [Python SDK|https://github.com/apache/beam/blob/a99c6826a067f49ebb60e625c8652900c7d0e810/sdks/python/apache_beam/io/gcp/bigquery.py#L1593], but not the Java SDK.

Although Java is more strict about types and schemas, the BigQueryIO transform supports writing TableRows which don't inherently have a schema. This provides a convenient path for loading JSON data into BigQuery but is massively thwarted by the fact that a schema is required to make use of the SchemaUpdateOption values ALLOW_FIELD_ADDITION and ALLOW_FIELD_RELAXATION.

The BigQuery schema autodetection feature must be enabled at the JobConfigurationLoad level. The BigQueryIO creates the JobConfigurationLoad in only one place: [WriteTables.java|https://github.com/apache/beam/blob/752bdfd09bc4175dd9f51a096f81c9e5b0805913/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L370]. Exposing the autodetection option would mean adding it here, then propagating the change upwards until it's exposed at the BigQueryIO.Write level.

A big of context on this issue:
 * Google cloud's blog [has an article|https://cloud.google.com/blog/products/gcp/how-to-handle-mutating-json-schemas-in-a-streaming-pipeline-with-square-enix] on handling mutating JSON schemas in Dataflow using a black-box "Validate and Mutate BQ Schema" step.
 * Suggested workarounds include creating a stateful DoFn to dynamically generate a schema, load it as a side input to create a PCollectionView, then passing it to BigQuerIO using withSchemaFromView: https://stackoverflow.com/a/58809875/477563
 * [Entire projects|https://github.com/the-dagger/dataflow-dynamic-schema] have been created to try and work around this issue.

All of the above would be rendered moot (and many headaches spared!) if only the schema autodetection were exposed in the Java SDK _like it already is_ in the Python SDK.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)