You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/02 03:20:42 UTC

[GitHub] [airflow] casassg opened a new issue #8059: [AIP-31] Enable customized XCom class

casassg opened a new issue #8059: [AIP-31] Enable customized XCom class
URL: https://github.com/apache/airflow/issues/8059
 
 
   
   **Description**
   
   
   
   - Enable custom XCom backend functionality for custom serialization/deserialization.
   - XCom backend can be set from airflow.cfg and defaults to existing `airflow.models.XCom`. 
   
   
   
   **Use case / motivation**
   This comes from a discussion with @turbaszek . We were talking that it would be nice to have a flexible XCom backend that allowed us to serialize/deserialize XCom values using custom logic. A possible extension here would be to enable large objects to be serialized into a custom file system while XCom would only include a pointer (and metadata) of the object in the file system. 
   
   An example use case would be data processing:
   
   ```
   
   @task
   def process_data(input: pd.DataFrame) -> pd.DataFrame
      # make some magic
   ```
   
   Where DataFrame are saved between tasks as `csv`.
   
   This is probably a stretch and maybe out of scope for this AIP (or require a new AIP). But wanted to put this out there.
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class

Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class
URL: https://github.com/apache/airflow/issues/8059#issuecomment-607664835
 
 
   Thanks @casassg for putting this into an issue! 
    
   The idea comes from DAGs where we have two operators:
   - OP1 that does: download from API and upload file on GCS
   - OP2 that does: download from GCS and do something
   
   Abstracting this layer of writing to and reading from persistent storage will make writing functional operators much easier. Because then my function will be called `process_data` not `download_and_process_data`.  That's why I think it's related to UX of functionals operators
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class

Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class
URL: https://github.com/apache/airflow/issues/8059#issuecomment-607664835
 
 
   Thanks @casassg for putting this into an issue! 
    
   The idea comes from DAGs where we have two operators OP1 >> OP2:
   - OP1 downloads from API and upload file on GCS
   - OP2 downloads the previous file from GCS and do something
   
   Abstracting this layer of writing to and reading from persistent storage will make writing functional operators much easier. Because then my function will be called `process_data` not `download_and_process_data`.  That's why I think it's related to UX of functionals operators
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on issue #8059: [AIP-31] Enable arbitrary XCom class

Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #8059: [AIP-31] Enable arbitrary XCom class
URL: https://github.com/apache/airflow/issues/8059#issuecomment-607664835
 
 
   Thanks @casassg for putting this into an issue! 
    
   The idea comes from DAGs where we have two operators:
   - OP1 that does: upload from API and put file on GCS
   - OP2 that does: download from GCS and do something
   
   Abstracting this layer of writing to and reading from persistent storage will make writing functional operators much easier. Because then my function will be called `process_data` not `download_and_process_data`.  That's why I think it's related to UX of functionals operators
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class

Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class
URL: https://github.com/apache/airflow/issues/8059#issuecomment-607664835
 
 
   Thanks @casassg for putting this into an issue! 
    
   The idea comes from DAGs where we have two operators OP1 >> OP2:
   - OP1 that does: download from API and upload file on GCS
   - OP2 that does: download from GCS and do something
   
   Abstracting this layer of writing to and reading from persistent storage will make writing functional operators much easier. Because then my function will be called `process_data` not `download_and_process_data`.  That's why I think it's related to UX of functionals operators
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class

Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #8059: [AIP-31] Enable arbitrary XCom class
URL: https://github.com/apache/airflow/issues/8059#issuecomment-607664835
 
 
   Thanks @casassg for putting this into an issue! 
    
   The idea comes from DAGs where we have two operators OP1 >> OP2:
   - OP1 downloads from API and upload file on GCS
   - OP2 downloads the previous file from GCS and does something
   
   Abstracting this layer of writing to and reading from persistent storage will make writing functional operators much easier. Because then my function will be called `process_data` not `download_and_process_data`.  That's why I think it's related to UX of functionals operators
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services