[GitHub] [airflow] maroshmka opened a new issue #12520: gcs_to_bq: Support schema in different bucket

[GitBox <gi...@apache.org>]

maroshmka opened a new issue #12520:
URL: https://github.com/apache/airflow/issues/12520


   **Description**
   
   Schema path (schema_object) for table in `GCSToBigQueryOperator` is downloading schema from source bucket. 
   
   New feature should support support also pulling schemas from different bucket.
   
   **Use case / motivation**
   
   I guess there are lot of use-cases. To mention one concrete:
   
   - we want to have schemas at 1 place, for us its terraform, where we define all resources and ACLs
   - those schemas are exported to GCS by CI
   - we want to use them in loads to BQ tables
   
   In case there is a schema change, we make 1 change (in terraform) and it works. Otherwise we would need to do terraform change, apply, then either update `schema_fields` in operator and release it or to allow adding fields on table load (which we don't necessarily want all the time).
   
   **Related Issues**
   
   Not directly, but at least - https://github.com/apache/airflow/issues/8280 - polish system tests to operator while at it


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] maroshmka commented on issue #12520: gcs_to_bq: Support schema in different bucket

[GitBox <gi...@apache.org>]

maroshmka commented on issue #12520:
URL: https://github.com/apache/airflow/issues/12520#issuecomment-731446573


   Implemented draft, if its ok to have this feature, I can finish the system tests :) 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org