[jira] [Commented] (AIRFLOW-5334) After Airflow upgrade to 1.10.3 in the gcs to bq operator it is mandatory to specify either autodetect or schema paramaters

["nexoriv (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057747#comment-17057747 ] 

nexoriv commented on AIRFLOW-5334:
----------------------------------

If you have the schema defined why do you need to set the schema_fields?

> After Airflow upgrade to 1.10.3 in the gcs to bq operator it is mandatory to specify either autodetect or schema paramaters
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-5334
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5334
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: gcp
>    Affects Versions: 1.10.3
>            Reporter: Benedetto Vitale
>            Priority: Major
>
> In PR [https://github.com/apache/airflow/pull/3880] the following check was introduced when adding support for autodetect feature in the GCS To BQ operator:
> {code:java}
> if schema_fields is None and not autodetect: 
>       raise ValueError( 'You must either pass a schema or autodetect=True.')
> {code}
> But this behaviour is supported at the moment by google apis themselves, so it indeed added a functionality this PR but also removed one implicitly, was this expected ?
> In our case we have big query schemas already defined with Google Deployment Manager and we do not need to pass either of those fields, is this a behaviour you guys consider a bug or is there something wrong with defining the schema outside Airflow itself according to you ? At the moment the 1.10.3 upgrade breaks many of our workflows because of that check (a similar check was added also to the Big Query hook)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)