You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Aaron Liblong (JIRA)" <ji...@apache.org> on 2019/06/17 17:56:00 UTC

[jira] [Commented] (BEAM-7570) withCustomGcsTempLocation should also be implemented for BigQueryIO.Read

    [ https://issues.apache.org/jira/browse/BEAM-7570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865826#comment-16865826 ] 

Aaron Liblong commented on BEAM-7570:
-------------------------------------

I've been looking into writing a patch for this ([all I have so far is some boilerplate|https://github.com/aliblong/beam/tree/bq-read-custom-gcs-temp/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery]), and I'm not sure how to go about making a ValueProvider accessible to the relevant functions. In BigQueryIO.Write, the function concerned with BQ temp location is part of a [class extending PTransform|https://github.com/aliblong/beam/blob/bq-read-custom-gcs-temp/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java#L78], so the ValueProvider can be passed in and get() called; this is not the case for BigQueryIO.Read, where I believe the relevant functions are [here|https://github.com/aliblong/beam/blob/bq-read-custom-gcs-temp/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java#L109], [here|https://github.com/aliblong/beam/blob/bq-read-custom-gcs-temp/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java#L141], and [here|https://github.com/aliblong/beam/blob/bq-read-custom-gcs-temp/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L849]. I don't have a strong understanding of the limitations of ValueProvider, so some of this may be wrong.

> withCustomGcsTempLocation should also be implemented for BigQueryIO.Read
> ------------------------------------------------------------------------
>
>                 Key: BEAM-7570
>                 URL: https://issues.apache.org/jira/browse/BEAM-7570
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-gcp
>            Reporter: Aaron Liblong
>            Priority: Major
>
> A function in BigQueryIO.Write called withCustomGcsTempLocation allows specification at template execution time of a GCS location used by BigQuery to write temp files. BigQuery also needs to write temp files for _read_ operations, and therefore this function should be available in BigQueryIO.Read.
> This issue blocks the ability to deploy a template with BigQuery read ops to an environment where users (who will execute the template) have only read access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)