[jira] [Commented] (AIRFLOW-4722) Add third PV mount to worker node to share data

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854681#comment-16854681 ] 

ASF GitHub Bot commented on AIRFLOW-4722:
-----------------------------------------

rummens commented on pull request #5366: [AIRFLOW-4722] Add third PV mount to worker node to share data
URL: https://github.com/apache/airflow/pull/5366
 
 
   For many use cases it is neccessary to write data from a worker back to a PV to share it between other workers (e.g. for TFX). Therefore a new (third) PVC should be optionaly mounted into the worker nodes (as well as the webserver). The PVC has to be provided by the user (similiar to DAG and log PV). It has to be mounted with the ReadOnly flag off. 
   
   As this is very new to me, I am not sure if I did everything correct. I was not able to run all tests but it should not break anything as this is mainly a copy of the log PV. Also I am not sure if the mount also works for the webserver pod.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW-4722)
   
   ### Description
   
   - Add a new parameter to add a new PV called airflow-data, which is mounted as writeable into a worker to allow sharing of data between workers.
   
   ### Tests
   
   - Sorry but I have no idea what to here :-/
   
   ### Documentation
   
   Updaded the default config with the new param. 
   
   ### Code Quality
   
   - How can I run this?
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Add third PV mount to worker node to share data
> -----------------------------------------------
>
>                 Key: AIRFLOW-4722
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4722
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: configuration, scheduler, worker
>    Affects Versions: 1.10.2, 1.10.3
>            Reporter: Marcel Rummens
>            Priority: Major
>   Original Estimate: 2m
>  Remaining Estimate: 2m
>
> For many use cases it is neccessary to write data from a worker back to a PV to share it between other workers (e.g. for TFX). Therefore a new (third) PVC should be optionaly mounted into the worker nodes (as well as the webserver). The PVC has to be provided by the user (similiar to DAG and log PV). It has to be mounted with the ReadOnly flag off. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)