You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kaxil Naik (JIRA)" <ji...@apache.org> on 2018/09/03 22:49:00 UTC
[jira] [Resolved] (AIRFLOW-398) Option to use cgroups to limit task
resource utilization
[ https://issues.apache.org/jira/browse/AIRFLOW-398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kaxil Naik resolved AIRFLOW-398.
--------------------------------
Resolution: Fixed
Was Resolved by https://github.com/apache/incubator-airflow/pull/1934
> Option to use cgroups to limit task resource utilization
> --------------------------------------------------------
>
> Key: AIRFLOW-398
> URL: https://issues.apache.org/jira/browse/AIRFLOW-398
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
> Reporter: Paul Yang
> Assignee: Paul Yang
> Priority: Minor
>
> Now that tasks have resource requirements specified, Airflow can use the defined resource requirements as limits when running the Airflow tasks. A common use case is to limit the memory used by a task so that it doesn't cause the worker machine to go OOM.
> The proposed idea is to implement a TaskRunner class that, by default, executes the Airflow run command in a bash subprocess as it is currently done now.
> Alternatively, the user can specify a TaskRunner class that uses cgroups to ensure that resource requirements are respected. This would be a lightweight approach to limiting resource usage by tasks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)