You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Aaron Liblong (JIRA)" <ji...@apache.org> on 2019/06/17 17:56:00 UTC
[jira] [Commented] (BEAM-7570) withCustomGcsTempLocation should
also be implemented for BigQueryIO.Read
[ https://issues.apache.org/jira/browse/BEAM-7570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865826#comment-16865826 ]
Aaron Liblong commented on BEAM-7570:
-------------------------------------
I've been looking into writing a patch for this ([all I have so far is some boilerplate|https://github.com/aliblong/beam/tree/bq-read-custom-gcs-temp/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery]), and I'm not sure how to go about making a ValueProvider accessible to the relevant functions. In BigQueryIO.Write, the function concerned with BQ temp location is part of a [class extending PTransform|https://github.com/aliblong/beam/blob/bq-read-custom-gcs-temp/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java#L78], so the ValueProvider can be passed in and get() called; this is not the case for BigQueryIO.Read, where I believe the relevant functions are [here|https://github.com/aliblong/beam/blob/bq-read-custom-gcs-temp/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java#L109], [here|https://github.com/aliblong/beam/blob/bq-read-custom-gcs-temp/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java#L141], and [here|https://github.com/aliblong/beam/blob/bq-read-custom-gcs-temp/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L849]. I don't have a strong understanding of the limitations of ValueProvider, so some of this may be wrong.
> withCustomGcsTempLocation should also be implemented for BigQueryIO.Read
> ------------------------------------------------------------------------
>
> Key: BEAM-7570
> URL: https://issues.apache.org/jira/browse/BEAM-7570
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Aaron Liblong
> Priority: Major
>
> A function in BigQueryIO.Write called withCustomGcsTempLocation allows specification at template execution time of a GCS location used by BigQuery to write temp files. BigQuery also needs to write temp files for _read_ operations, and therefore this function should be available in BigQueryIO.Read.
> This issue blocks the ability to deploy a template with BigQuery read ops to an environment where users (who will execute the template) have only read access.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)