You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/09 16:40:16 UTC

[GitHub] [airflow] jimmajure opened a new issue #9181: Make KubernetesExecutor PersistentVolumeClaim mounts more flexible.

jimmajure opened a new issue #9181:
URL: https://github.com/apache/airflow/issues/9181


   
   **Description**
   
   In the KubernetesExecutor configuration, allow the DAGs, logs, plugins folders to be supported by a single PersistentVolumeClaim. Currently the configuration forces a separate PVC for each directory. 
   
   **Use case / motivation**
   
   Mounting volumes via a PVC is an expensive task that requires multiple calls to the K8S API server, so it is beneficial to be able to mount fewer PVC volumes for each pod. We have run into scalability issues in AWS EKS when using EFS drives to support DAGs and logs. 
   
   The configuration should allow users to mount as many PVC mounts as required. If people choose to put DAGs, logs and plugins in a single PVC mount, that should be supported. If they choose to put each in a separate PVC mount, that can be supported also. 
   
   This should be able to be supported by separating the purpose of the mount from the mount, itself. Simply allow one or more mounts and let the user map the directories to the appropriate folder, irrespective of which mount it is on.
   
   **Related Issues**
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #9181: Make KubernetesExecutor PersistentVolumeClaim mounts more flexible.

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on issue #9181:
URL: https://github.com/apache/airflow/issues/9181#issuecomment-1001260281


   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9181: Make KubernetesExecutor PersistentVolumeClaim mounts more flexible.

Posted by GitBox <gi...@apache.org>.

boring-cyborg[bot] commented on issue #9181:
URL: https://github.com/apache/airflow/issues/9181#issuecomment-640645906


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj edited a comment on issue #9181: Make KubernetesExecutor PersistentVolumeClaim mounts more flexible.

Posted by GitBox <gi...@apache.org>.

mik-laj edited a comment on issue #9181:
URL: https://github.com/apache/airflow/issues/9181#issuecomment-671089535


   Can the expected result be achieved with pod_template_file option?
   https://airflow.readthedocs.io/en/latest/configurations-ref.html#pod-template-file


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #9181: Make KubernetesExecutor PersistentVolumeClaim mounts more flexible.

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on issue #9181:
URL: https://github.com/apache/airflow/issues/9181#issuecomment-996289403


   This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed issue #9181: Make KubernetesExecutor PersistentVolumeClaim mounts more flexible.

Posted by GitBox <gi...@apache.org>.

github-actions[bot] closed issue #9181:
URL: https://github.com/apache/airflow/issues/9181


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #9181: Make KubernetesExecutor PersistentVolumeClaim mounts more flexible.

Posted by GitBox <gi...@apache.org>.

mik-laj commented on issue #9181:
URL: https://github.com/apache/airflow/issues/9181#issuecomment-671089535


   Can the expected result be achieved with this configuration option?
   https://airflow.readthedocs.io/en/latest/configurations-ref.html#pod-template-file


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org