You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "benjyblack (via GitHub)" <gi...@apache.org> on 2023/03/03 23:29:17 UTC

[GitHub] [airflow] benjyblack opened a new issue, #29912: BigQueryToGCSOperator does not wait for completion

benjyblack opened a new issue, #29912:
URL: https://github.com/apache/airflow/issues/29912

   ### Apache Airflow Provider(s)
   
   google
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-google==7.0.0
   
   ### Apache Airflow version
   
   2.3.2
   
   ### Operating System
   
   Debian GNU/Linux
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   [Deferrable mode for BigQueryToGCSOperator #27683](https://github.com/apache/airflow/pull/27683) changed the functionality of the `BigQueryToGCSOperator` so that it no longer waits for the completion of the operation. This is because the `nowait=True` parameter is now [being set](https://github.com/apache/airflow/pull/27683/files#diff-23c5b2e773487f9c28b75b511dbf7269eda1366f16dec84a349d95fa033ffb3eR191).
   
   ### What you think should happen instead
   
   This is unexpected behavior. Any downstream tasks of the `BigQueryToGCSOperator` that expect the CSVs to have been written by the time they are called may result in errors (and have done so in our own operations).
   
   The property should at least be configurable.
   
   ### How to reproduce
   
   1. Leverage the `BigQueryToGcsOperator` in your DAG.
   2. Have it write a large table to a CSV somewhere in GCS
   3. Notice that the task completes almost immediately but the CSVs may not exist in GCS until later.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #29912: BigQueryToGCSOperator does not wait for completion

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29912: BigQueryToGCSOperator does not wait for completion
URL: https://github.com/apache/airflow/issues/29912


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29912: BigQueryToGCSOperator does not wait for completion

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29912:
URL: https://github.com/apache/airflow/issues/29912#issuecomment-1454260539

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hussein-awala commented on issue #29912: BigQueryToGCSOperator does not wait for completion

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #29912:
URL: https://github.com/apache/airflow/issues/29912#issuecomment-1454965669

   @benjyblack Sorry I just tested a solution to fix this bug and it worked, so I created the PR #29925, can you check it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hussein-awala commented on issue #29912: BigQueryToGCSOperator does not wait for completion

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #29912:
URL: https://github.com/apache/airflow/issues/29912#issuecomment-1454323163

   @benjyblack I assigned it to you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org