You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/02 03:20:42 UTC
[GitHub] [airflow] casassg opened a new issue #8059: [AIP-31] Enable
customized XCom class
casassg opened a new issue #8059: [AIP-31] Enable customized XCom class
URL: https://github.com/apache/airflow/issues/8059
**Description**
- Enable custom XCom backend functionality for custom serialization/deserialization.
- XCom backend can be set from airflow.cfg and defaults to existing `airflow.models.XCom`.
**Use case / motivation**
This comes from a discussion with @turbaszek . We were talking that it would be nice to have a flexible XCom backend that allowed us to serialize/deserialize XCom values using custom logic. A possible extension here would be to enable large objects to be serialized into a custom file system while XCom would only include a pointer (and metadata) of the object in the file system.
An example use case would be data processing:
```
@task
def process_data(input: pd.DataFrame) -> pd.DataFrame
# make some magic
```
Where DataFrame are saved between tasks as `csv`.
This is probably a stretch and maybe out of scope for this AIP (or require a new AIP). But wanted to put this out there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek edited a comment on issue #8059: [AIP-31]
Enable arbitrary XCom class
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class
URL: https://github.com/apache/airflow/issues/8059#issuecomment-607664835
Thanks @casassg for putting this into an issue!
The idea comes from DAGs where we have two operators:
- OP1 that does: download from API and upload file on GCS
- OP2 that does: download from GCS and do something
Abstracting this layer of writing to and reading from persistent storage will make writing functional operators much easier. Because then my function will be called `process_data` not `download_and_process_data`. That's why I think it's related to UX of functionals operators
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek edited a comment on issue #8059: [AIP-31]
Enable arbitrary XCom class
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class
URL: https://github.com/apache/airflow/issues/8059#issuecomment-607664835
Thanks @casassg for putting this into an issue!
The idea comes from DAGs where we have two operators OP1 >> OP2:
- OP1 downloads from API and upload file on GCS
- OP2 downloads the previous file from GCS and do something
Abstracting this layer of writing to and reading from persistent storage will make writing functional operators much easier. Because then my function will be called `process_data` not `download_and_process_data`. That's why I think it's related to UX of functionals operators
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on issue #8059: [AIP-31] Enable
arbitrary XCom class
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #8059: [AIP-31] Enable arbitrary XCom class
URL: https://github.com/apache/airflow/issues/8059#issuecomment-607664835
Thanks @casassg for putting this into an issue!
The idea comes from DAGs where we have two operators:
- OP1 that does: upload from API and put file on GCS
- OP2 that does: download from GCS and do something
Abstracting this layer of writing to and reading from persistent storage will make writing functional operators much easier. Because then my function will be called `process_data` not `download_and_process_data`. That's why I think it's related to UX of functionals operators
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek edited a comment on issue #8059: [AIP-31]
Enable arbitrary XCom class
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class
URL: https://github.com/apache/airflow/issues/8059#issuecomment-607664835
Thanks @casassg for putting this into an issue!
The idea comes from DAGs where we have two operators OP1 >> OP2:
- OP1 that does: download from API and upload file on GCS
- OP2 that does: download from GCS and do something
Abstracting this layer of writing to and reading from persistent storage will make writing functional operators much easier. Because then my function will be called `process_data` not `download_and_process_data`. That's why I think it's related to UX of functionals operators
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek edited a comment on issue #8059: [AIP-31]
Enable arbitrary XCom class
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class
URL: https://github.com/apache/airflow/issues/8059#issuecomment-607664835
Thanks @casassg for putting this into an issue!
The idea comes from DAGs where we have two operators OP1 >> OP2:
- OP1 downloads from API and upload file on GCS
- OP2 downloads the previous file from GCS and does something
Abstracting this layer of writing to and reading from persistent storage will make writing functional operators much easier. Because then my function will be called `process_data` not `download_and_process_data`. That's why I think it's related to UX of functionals operators
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services