[GitHub] [airflow] sighingnow commented on pull request #17405: Add XCom.clear so it's hookable in custom XCom backend

[GitBox <gi...@apache.org>]

sighingnow commented on pull request #17405:
URL: https://github.com/apache/airflow/pull/17405#issuecomment-929962066


   Hi @MatrixManAtYrService, I would like to share our experiences of implementing the airflow XCom backend in vineyard (see also: https://github.com/v6d-io/v6d/blob/main/python/vineyard/contrib/airflow/xcom/backend.py)
   
   Background: vineyard a in-memory storage that that share complex composable data objects with clients in a zero-copy fashion using memory sharing.
   
   We implements a XCom backend based on vineyard. Each object in vineyard has own unique identifier (`ObjectID`), then
   
   + for serialization, we first put the data into vineyard, then store the returned object id into the db of airflow
   + for deserailization, we implemented as first retrieve the object id, then getting object from vineyard
   + for deletion, we first query the object id for required `(dag, task, execution_date)` then deleting it from vineyard.
   
   Hope that could be helpful for you. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org