You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/21 09:35:21 UTC

[GitHub] [airflow] TY-chang opened a new issue, #24578: use insert_job to execute gcs_to_bq_operator

TY-chang opened a new issue, #24578:
URL: https://github.com/apache/airflow/issues/24578

   ### Description
   
   - I met LoadJob bug in `BigQueryInsertJobOperator` and tried to use `GCSToBigQueryOperator` instead when update from Airflow 2.2.5 to 2.3.2.
   
   
   ### Use case/motivation
   
   - most BigQuery-related operator recommend that we should use `insert_job()`, but gcs_to_bq_operator still use `run_load ()`.
   - `GCSToBigQueryOperator` still uses lots of parameter to build config. Not like `BigQueryInsertJobOperator` which uses a clean configuration. The configuration is much straightforward and easy to follow.
   - I would like to know whether to  keep `GCSToBigQueryOperator` and replace `run_load` with `insert_job`, Or have `BigQueryInsertJobOperator`. If we could replace `run_load` with `insert_job` in `GCSToBigQueryOperator` , all BigQuery-related operator could be more consistent.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #24578: use insert_job to execute gcs_to_bq_operator

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24578:
URL: https://github.com/apache/airflow/issues/24578#issuecomment-1161504547

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] TY-chang commented on issue #24578: use insert_job to execute gcs_to_bq_operator

Posted by GitBox <gi...@apache.org>.
TY-chang commented on issue #24578:
URL: https://github.com/apache/airflow/issues/24578#issuecomment-1161507419

   Here is error I met when we updated to` version 2.3.2`:
   ```
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/operators/bigquery.py", line 2170, in execute
       table = job.to_api_repr()["configuration"]["query"]["destinationTable"]
   KeyError: 'query'
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #24578: use insert_job to execute gcs_to_bq_operator

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #24578:
URL: https://github.com/apache/airflow/issues/24578#issuecomment-1161715172

   duplicate: https://github.com/apache/airflow/issues/23826
   Please update your google provider.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #24578: use insert_job to execute gcs_to_bq_operator

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #24578: use insert_job to execute gcs_to_bq_operator
URL: https://github.com/apache/airflow/issues/24578


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org