You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Jonathan Sulman (Jira)" <ji...@apache.org> on 2020/04/22 17:05:00 UTC
[jira] [Updated] (BEAM-9804) beam.io.BigQuerySource needs
permissions to create datasets to be able to run queries
[ https://issues.apache.org/jira/browse/BEAM-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Sulman updated BEAM-9804:
----------------------------------
Description:
Based on BEAM-8458, which was closed with a Java fix in 2.20.0. However, the bug still exists in the python SDK.
When using BigQuerySource with the query option, BigQueryReader creates a temporary dataset to store the results of the query.
Therefore, Beam requires permissions to create datasets just to be able to run a query. In practice, this means that Beam requires the role bigQuery.User just to run queries, whereas if you use {{from}} (to read from a table), the role bigQuery.jobUser suffices.
BigqueryDataSource should have an option to set an existing dataset to write the temp results of
a query, so it would be enough with having the role bigQuery.jobUser.
was:
Based on BEAM-8458, which was closed with a Java fix in 2.20.0. However, the bug still exists in the python SDK.
When using {{BigQuerySource with the query option, BigQueryReader }}creates a temporary dataset to store the results of the query.
Therefore, Beam requires permissions to create datasets just to be able to run a query. In practice, this means that Beam requires the role bigQuery.User just to run queries, whereas if you use {{from}} (to read from a table), the role bigQuery.jobUser suffices.
BigqueryDataSource should have an option to set an existing dataset to write the temp results of
a query, so it would be enough with having the role bigQuery.jobUser.
> beam.io.BigQuerySource needs permissions to create datasets to be able to run queries
> -------------------------------------------------------------------------------------
>
> Key: BEAM-9804
> URL: https://issues.apache.org/jira/browse/BEAM-9804
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
> Reporter: Jonathan Sulman
> Priority: Major
>
> Based on BEAM-8458, which was closed with a Java fix in 2.20.0. However, the bug still exists in the python SDK.
> When using BigQuerySource with the query option, BigQueryReader creates a temporary dataset to store the results of the query.
> Therefore, Beam requires permissions to create datasets just to be able to run a query. In practice, this means that Beam requires the role bigQuery.User just to run queries, whereas if you use {{from}} (to read from a table), the role bigQuery.jobUser suffices.
> BigqueryDataSource should have an option to set an existing dataset to write the temp results of
> a query, so it would be enough with having the role bigQuery.jobUser.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)