You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Hongyi Wang (Jira)" <ji...@apache.org> on 2020/04/01 18:44:00 UTC
[jira] [Assigned] (AIRFLOW-7117) Honor self.schema in sql_to_gcs as
schema to upload
[ https://issues.apache.org/jira/browse/AIRFLOW-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hongyi Wang reassigned AIRFLOW-7117:
------------------------------------
Assignee: Hongyi Wang
> Honor self.schema in sql_to_gcs as schema to upload
> ---------------------------------------------------
>
> Key: AIRFLOW-7117
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7117
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
> Affects Versions: 1.10.5, 1.10.6, 1.10.7, 1.10.8, 1.10.9
> Reporter: Hongyi Wang
> Assignee: Hongyi Wang
> Priority: Minor
>
> In [sql_to_gcs|https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/sql_to_gcs.py], there is a [self.schema|https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/sql_to_gcs.py#L61-L65], which was introduced in [pr-3091|https://github.com/apache/airflow/pull/3091/files] to allow user to provide their own schema while loading from sql to Google BigQuery.
> This is very useful when user want custom type mapping. For example, [mysql.date is default to map bigquery.TIMESTAMP|https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/mysql_to_gcs.py#L49], but sometimes, people want it to be bigquery.DATE.
> However, during [mysql_to_gcs|https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/mysql_to_gcs.py] re-organization, we lost that ability. (Only [use self.schema for pre-processing|https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/sql_to_gcs.py#L229-L232], while data type is no longer configurable).
> As a conclusion, we need to honor self.schema (if provided) to override default schema, like we did before.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)