[GitHub] [airflow] eladkal commented on issue #13898: S3 to GCS to BQ daily incremental data transfer pipeline. Append data from only the incremental (newly added) files from GCS into BQ table?

[GitBox <gi...@apache.org>]

eladkal commented on issue #13898:
URL: https://github.com/apache/airflow/issues/13898#issuecomment-768041034


   Xcom has limits. In Airflow 2 you can define a custom xcom backend 
   see [documentation](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html?highlight=xcom#custom-xcom-backend), [polidea article](https://www.polidea.com/blog/airflow-2-0-dag-authoring-redesigned/)
   
   I don't really know your use case but I'm not really sure if xcom are even needed here. Why not simply writing the delta (increment) to a folder in GCS in a known path? For example using `{{ ds }}` for daily runs or  `{{ ts_nodash }}` for hourly runs then the loading job can just read the data from this known path without looking in xcoms.
   
   In any case since this is not a bug/feature request but more of a troubleshooting question for a specific ETL I'm closing this issue. If you need further assistance please ask on https://stackoverflow.com/ 
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org