You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/01 11:14:32 UTC
[GitHub] [airflow] msumit opened a new pull request #12733: WIP: RedisXCom backend
msumit opened a new pull request #12733:
URL: https://github.com/apache/airflow/pull/12733
Testing out the approach to add a Redis backend for XCom. Will add unit tests and documentation if the approach looks fine.
Tested the functionality of XComs via `example_xcom` dag and from UI as well.
cc - @vikramkoka @kaxil
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #12733: WIP: RedisXCom backend
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #12733:
URL: https://github.com/apache/airflow/pull/12733#issuecomment-739438331
Just catching up here, good work @msumit . I was thinking we could add this kind of backends as example_redis_xcom_backend similar to our example_dags which we can use for the docs too.
I am happy to help in this effort. It might have to wait post 2.0rc though
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12733: WIP: RedisXCom backend
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12733:
URL: https://github.com/apache/airflow/pull/12733#issuecomment-737215744
[The Workflow run](https://github.com/apache/airflow/actions/runs/396098452) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #12733: WIP: RedisXCom backend
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #12733:
URL: https://github.com/apache/airflow/pull/12733#issuecomment-737213320
> To be frank I could not find much info on this topic except [this](https://airflow.apache.org/docs/stable/concepts.html#custom-xcom-backend), which doesn't reveal much
Yesterday we published a blog with [an example](https://www.polidea.com/blog/airflow-2-0-dag-authoring-redesigned/#custom-xcom-backends-8560) where users can store anything between tasks. True, it's not official documentation.
I was thinking about custom XCom backends since we implemented them (already > 6 months) and I lean toward creating a guide / examples in docs instead of ready to use code. In my opinion, this feature works best when is cut to users need. I happy to write one if we decide this is a way to go.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on pull request #12733: WIP: RedisXCom backend
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on pull request #12733:
URL: https://github.com/apache/airflow/pull/12733#issuecomment-737374944
Nice work indeed @msumit !
As I shared in the mail list thread started by @turbaszek , personally I would agree with him to create better documentation around the custom XCom feature instead of accepting more custom XCom into the code base.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek edited a comment on pull request #12733: WIP: RedisXCom backend
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on pull request #12733:
URL: https://github.com/apache/airflow/pull/12733#issuecomment-737201582
@msumit nice work! However, I think that we first have to agree as a community if we want to maintain custom XCom backends. Those can be either simple or complex code.
In my opinion users already have most of the tools to build XCom backends that are suited their needs (for example plenty of hooks) and they can build solution that answer 100% of their needs. One of the ideas behind XComs is not only to use "other storage" but also automate logic of handling non-serialisable data types like `pandas.DataFrame` which can be passes between tasks. This one step further in making Airflow more data-aware.
Also, adding more custom stuff to Airflow core in my opinion is not a best idea - community will have to maintain it.
If we decide to have XCom backends in main repo, then we should:
- define how generic or problem oriented should the backends be
- make sure it's easy to extend them to adjust to users needs
- all have proper documentation and tests
What others think? @ashb @kaxil @potiuk @mik-laj @XD-DENG
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] msumit closed pull request #12733: WIP: RedisXCom backend
Posted by GitBox <gi...@apache.org>.
msumit closed pull request #12733:
URL: https://github.com/apache/airflow/pull/12733
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] msumit commented on pull request #12733: WIP: RedisXCom backend
Posted by GitBox <gi...@apache.org>.
msumit commented on pull request #12733:
URL: https://github.com/apache/airflow/pull/12733#issuecomment-737209891
@turbaszek thanks for the comments. I just wanted to see how easy or difficult is to write a custom XCom backend and Redis backend seems to be a good start. To be frank I could not find much info on this topic except [this](https://airflow.apache.org/docs/stable/concepts.html#custom-xcom-backend), which doesn't reveal much.
I won't mind discarding this PR as it was just some testing ground, but IMO even a Redis backend is a significant improvement over the DB based backend, as users would be able to store upto 512mbs of data vs some kbs at present. Also, it'll help as a blueprint on how to extend the BaseXCom to their needs.
Also, I'm not sure who is assigned to manage all other backported modules, AFAIK except some big SAAS companies, we (the community) would only be managing them, but I could be wrong here, so would love to hear from others.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] msumit commented on pull request #12733: WIP: RedisXCom backend
Posted by GitBox <gi...@apache.org>.
msumit commented on pull request #12733:
URL: https://github.com/apache/airflow/pull/12733#issuecomment-738645692
Created a GIST, so it can help if someone searches for it. https://gist.github.com/msumit/1bc995c4a21acb257a39c36f0279e3d6
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #12733: WIP: RedisXCom backend
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #12733:
URL: https://github.com/apache/airflow/pull/12733#issuecomment-737201582
@msumit nice work! However, I think that we first have to agree as a community if we want to maintain custom XCom backends. Those can be either simple or complex code.
In my opinion users already have most of the tools to build XCom backends that are suited their needs (for example plenty of hooks) and they can build solution that answer 100% of their needs. Also, adding more custom stuff to Airflow core in my opinion is not a best idea - community will have to maintain it.
If we decide to have XCom backends in main repo, then we should:
- define how generic or problem oriented should the backends be
- make sure it's easy to extend them to adjust to users needs
- all have proper documentation and tests
What others think? @ashb @kaxil @potiuk @mik-laj @XD-DENG
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on pull request #12733: WIP: RedisXCom backend
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on pull request #12733:
URL: https://github.com/apache/airflow/pull/12733#issuecomment-738646245
Thanks @msumit !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org