You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/10/02 18:45:00 UTC
[jira] [Work logged] (BEAM-5457) BigQuerySource(query=...) in
DirectRunner creates temp dataset in the wrong location
[ https://issues.apache.org/jira/browse/BEAM-5457?focusedWorklogId=150495&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150495 ]
ASF GitHub Bot logged work on BEAM-5457:
----------------------------------------
Author: ASF GitHub Bot
Created on: 02/Oct/18 18:44
Start Date: 02/Oct/18 18:44
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #6463: [BEAM-5457] Make BigQuerySource work for the EU
URL: https://github.com/apache/beam/pull/6463#discussion_r222067746
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -661,8 +667,18 @@ def __init__(self, source, test_bigquery_client=None, use_legacy_sql=True,
def _get_source_table_location(self):
tr = self.source.table_reference
- if tr is None:
- # TODO: implement location retrieval for query sources
+ if tr is None: # it's a query source
+ if self.source.location is not None:
+ logging.info('Using location=%r from BigQuerySource',
+ self.source.location)
+ return self.source.location
+
+ # TODO(BEAM-5457): Is there a way to get the source location from a query?
Review comment:
I've investigated this in the past on another PR, and you can run a query in dry-run mode to get the list of referenced tables: https://github.com/apache/beam/pull/5435#discussion_r213817094
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 150495)
Time Spent: 0.5h (was: 20m)
> BigQuerySource(query=...) in DirectRunner creates temp dataset in the wrong location
> ------------------------------------------------------------------------------------
>
> Key: BEAM-5457
> URL: https://issues.apache.org/jira/browse/BEAM-5457
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.6.0
> Reporter: Joar Wandborg
> Assignee: Chamikara Jayalath
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm in the EU, if I have a
>
> {code:java}
> BigQuerySource(
> query="SELECT x, y FROM `my-other-project.mydataset.my_european_table`",
> project="myproject",
> use_standard_sql=True
> ){code}
> And then run the Pipeline through the DirectRunner I get the following warning and error:
> {noformat}
> 2018-09-21 11:39:52,620 WARNING root create_temporary_dataset
> Dataset myproject:temp_dataset_0bbb28f014a24225b668a67341f4f71e does not exist so we will create it as temporary with location=None {noformat}
> {noformat}
> HttpBadRequestError: HttpError accessing <https://www.googleapis.com/bigquery/v2/projects/myproject/queries/xyz123?alt=json&maxResults=10000>: response: <{'status': '400', 'content-length': '354', 'x-xss-protection': '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Fri, 21 Sep 2018 09:39:55 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'quic=":443"; ma=2592000; v="44,43,39,35"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
> "error": {
> "code": 400,
> "message": "Cannot read and write in different locations: source: EU, destination: US",
> "errors": [
> {
> "message": "Cannot read and write in different locations: source: EU, destination: US",
> "domain": "global",
> "reason": "invalid"
> }
> ],
> "status": "INVALID_ARGUMENT"
> }
> {noformat}
> There's a TODO in the code that looks very related: [https://github.com/apache/beam/blob/d691a86b8fd082efd0fd71c3cb58b7d61442717d/sdks/python/apache_beam/io/gcp/bigquery.py#L665|https://github.com/apache/beam/blob/d691a86b8fd082efd0fd71c3cb58b7d61442717d/sdks/python/apache_beam/io/gcp/bigquery.py#L665,]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)