[GitHub] [airflow] turbaszek edited a comment on pull request #12733: WIP: RedisXCom backend

[GitBox <gi...@apache.org>]

turbaszek edited a comment on pull request #12733:
URL: https://github.com/apache/airflow/pull/12733#issuecomment-737201582


   @msumit nice work! However, I think that we first have to agree as a community if we want to maintain custom XCom backends. Those can be either simple or complex code.
   
   In my opinion users already have most of the tools to build XCom backends that are suited their needs (for example plenty of hooks) and they can build solution that answer 100% of their needs. One of the ideas behind XComs is not only to use "other storage" but also automate logic of handling non-serialisable data types like `pandas.DataFrame` which can be passes between tasks. This one step further in making Airflow more data-aware. 
   
   Also, adding more custom stuff to Airflow core in my opinion is not a best idea - community will have to maintain it. 
   
   If we decide to have XCom backends in main repo, then we should:
   - define how generic or problem oriented should the backends be
   - make sure it's easy to extend them to adjust to users needs
   - all have proper documentation and tests
   
   What others think? @ashb @kaxil @potiuk @mik-laj @XD-DENG 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org