You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Ryan Yuan (JIRA)" <ji...@apache.org> on 2019/07/08 06:42:00 UTC
[jira] [Updated] (AIRFLOW-4908) Implement BigQuery Hooks/Operators
for update_dataset, patch_dataset and get_dataset
[ https://issues.apache.org/jira/browse/AIRFLOW-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan Yuan updated AIRFLOW-4908:
-------------------------------
Description:
To create a BigQuery sink for GCP Stackdriver Logging, I have to assign `WRITER` access to group `cloud-logs@google.com` to access BQ dataset. However, current BigQueryHook doesn't support updating/patching dataset.
Reference: [https://googleapis.github.io/google-cloud-python/latest/logging/usage.html#export-to-bigquery]
Implement GCP Stackdriver Logging: https://issues.apache.org/jira/browse/AIRFLOW-4779
While it is missing update_dataset and patch_dataset, BigQueryHook has get_dataset but it doesn't have operator for it.
Features to be implemented:
BigQueryBaseCursor.patch_dataset
BigQueryBaseCursor.update_dataset
BigQueryPatchDatasetOperator
BigQueryUpdateDatasetOperator
BigQueryGetDatasetOperator
was:
To create a BigQuery sink for GCP Stackdriver Logging, I have to assign `WRITER` access to group `cloud-logs@google.com` to access BQ dataset. However, current BigQueryHook doesn't support updating/patching dataset.
Reference: [https://googleapis.github.io/google-cloud-python/latest/logging/usage.html#export-to-bigquery]
Implement GCP Stackdriver Logging: [https://issues.apache.org/jira/browse/AIRFLOW-4779|https://issues.apache.org/jira/browse/AIRFLOW-4779]
Summary: Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (was: Implement BigQueryHook's update_dataset and patch_dataset)
> Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset
> ------------------------------------------------------------------------------------
>
> Key: AIRFLOW-4908
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4908
> Project: Apache Airflow
> Issue Type: New Feature
> Components: gcp
> Affects Versions: 2.0.0
> Reporter: Ryan Yuan
> Assignee: Ryan Yuan
> Priority: Critical
>
> To create a BigQuery sink for GCP Stackdriver Logging, I have to assign `WRITER` access to group `cloud-logs@google.com` to access BQ dataset. However, current BigQueryHook doesn't support updating/patching dataset.
> Reference: [https://googleapis.github.io/google-cloud-python/latest/logging/usage.html#export-to-bigquery]
> Implement GCP Stackdriver Logging: https://issues.apache.org/jira/browse/AIRFLOW-4779
> While it is missing update_dataset and patch_dataset, BigQueryHook has get_dataset but it doesn't have operator for it.
>
> Features to be implemented:
> BigQueryBaseCursor.patch_dataset
> BigQueryBaseCursor.update_dataset
> BigQueryPatchDatasetOperator
> BigQueryUpdateDatasetOperator
> BigQueryGetDatasetOperator
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)