You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/07 23:36:44 UTC

[GitHub] [airflow] patricker opened a new issue #22065: DB To GCS Operations Should Return/Save Row Count

patricker opened a new issue #22065:
URL: https://github.com/apache/airflow/issues/22065


   ### Description
   
   All DB to GCS Operators should track the per file and total row count written for metadata and validation purposes.
   
   - Optionally, based on param, include the row count metadata as GCS file upload metadata.
   - Always return row count data through XCom. Currently this operator has no return value.
   
   ### Use case/motivation
   
   Currently, there is no way to check the uploaded files row count without opening the file. Downstream operations should have access to this information, and allowing it to be saved as GCS metadata and returning it through XCom makes it readily available for other uses.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] patricker commented on issue #22065: DB To GCS Operations Should Return/Save Row Count

Posted by GitBox <gi...@apache.org>.
patricker commented on issue #22065:
URL: https://github.com/apache/airflow/issues/22065#issuecomment-1061973217


   @eladkal I actually wrote this up last night. I didn't have to touch any of the implementation specific classes. I only had to change `BaseSQLToGCSOperator`. Still testing the code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #22065: DB To GCS Operations Should Return/Save Row Count

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #22065:
URL: https://github.com/apache/airflow/issues/22065#issuecomment-1061464734


   This would be easier after https://github.com/apache/airflow/issues/21599


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org