You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/09 16:56:14 UTC

[GitHub] [airflow] mjwinkler opened a new issue, #28806: BaseSQLToGCSOperator no longer returns at least one file even if empty

mjwinkler opened a new issue, #28806:
URL: https://github.com/apache/airflow/issues/28806

   ### Apache Airflow Provider(s)
   
   google
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-google==8.6.0
   
   ### Apache Airflow version
   
   2.5.0
   
   ### Operating System
   
   Debian GNU/Linux
   
   ### Deployment
   
   Astronomer
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   PR `Expose SQL to GCS Metadata (https://github.com/apache/airflow/pull/24382)` made a breaking change [here](https://github.com/apache/airflow/blob/3eee33ac8cb74cfbb08bce9090e9c601cf98da44/airflow/providers/google/cloud/transfers/sql_to_gcs.py#L286) that results in no files being returned when there are no data rows (empty table) rather than a single empty file as in the past.
   
   ### What you think should happen instead
   
   I would like to preserve the original behavior of having at least one file returned even if it is empty. Or to make that behavior optional via a new parameter.
   
   The original behavior can be implemented with the following code change:
   FROM:
   ```
   if file_to_upload["file_row_count"] > 0:
               yield file_to_upload
   ```
   TO:
   ```
   if file_no == 0 or file_to_upload["file_row_count"] > 0:
               yield file_to_upload
   ```
   
   ### How to reproduce
   
   Create a DAG that uses BaseSQLToGCSOperator with a SQL command that references an empty SQL table or returns no rows. The `execute` method will not write any files.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] vchiapaikeo commented on issue #28806: BaseSQLToGCSOperator no longer returns at least one file even if empty

Posted by GitBox <gi...@apache.org>.
vchiapaikeo commented on issue #28806:
URL: https://github.com/apache/airflow/issues/28806#issuecomment-1385647686

   I can take this -
   
   https://github.com/apache/airflow/pull/28959 adds a write_on_empty param to BaseSQLToGCSOperator that will write an empty file if no rows are returned.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #28806: BaseSQLToGCSOperator no longer returns at least one file even if empty

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28806:
URL: https://github.com/apache/airflow/issues/28806#issuecomment-1375946321

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #28806: BaseSQLToGCSOperator no longer returns at least one file even if empty

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28806:
URL: https://github.com/apache/airflow/issues/28806#issuecomment-1376372942

   Marked it as good first issue (but If you are willing to implement it / add tests - the fastest way to get it implement it is to implement it yourself (just saying).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #28806: BaseSQLToGCSOperator no longer returns at least one file even if empty

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #28806: BaseSQLToGCSOperator no longer returns at least one file even if empty
URL: https://github.com/apache/airflow/issues/28806


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org