You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "David Sabater (JIRA)" <ji...@apache.org> on 2017/12/25 21:36:02 UTC
[jira] [Commented] (AIRFLOW-1943) Add feature to create External
BigQuery Table
[ https://issues.apache.org/jira/browse/AIRFLOW-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16303396#comment-16303396 ]
David Sabater commented on AIRFLOW-1943:
----------------------------------------
This can probably be implemented by reusing the current BigQuery Operator as the BigQuery hook accepts input parameters for the *configuration.query.tableDefinitions* property already, which are key to create external tables as per Google documentation:
1. The sourceUris must be fully-qualified, in the format gs://[BUCKET]/[OBJECT] using *configuration.query.tableDefinitions.(key).sourceUris[]*
2. Specify the rest of external data source properties by setting the *configuration.query.tableDefinitions* property
> Add feature to create External BigQuery Table
> ---------------------------------------------
>
> Key: AIRFLOW-1943
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1943
> Project: Apache Airflow
> Issue Type: Task
> Components: contrib, gcp
> Affects Versions: Airflow 2.0
> Reporter: Kaxil Naik
> Priority: Minor
> Fix For: Airflow 2.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Currently using the BigQuery operator and the bq hook we can just create a materialized (internal) BigQuery Table but there is no code to create an external BQ table.
> Below are the links to Python Api and the JSON Rest API that can be helpful:
> * https://github.com/GoogleCloudPlatform/google-cloud-python/commit/7ea861fce5023409b1f8c0baa1ea7d90b41a2300
> * https://github.com/GoogleCloudPlatform/google-cloud-python/commit/3af22d7fd595e2d16f274d59e34f7a592ae251b5
> * https://cloud.google.com/bigquery/external-table-definition#json_table_definitions_1
> * https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)