You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/21 13:01:52 UTC
[GitHub] [airflow] ecerulm opened a new issue #16563: executor_config: "limits_memory" option for operators
ecerulm opened a new issue #16563:
URL: https://github.com/apache/airflow/issues/16563
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
<!-- A short description of your feature -->
Currently, if you want to change the kubernetes resources allocated to an operator you need to something like
```
from kubernetes.client import models as k8s
task1 = PythonOperator(
task_id="id1",
python_callable=mycallable,
dag=dag1,
executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
limits={
"memory": "6Gi",
"cpu": "1",
}
)
)
]
)
)
},
```
which works, but I think that is very verbose and for a (in my opinion) common case. I would prefer a more direct syntax like
```
task1 = PythonOperator(
task_id="id1",
python_callable=mycallable,
dag=dag1,
executor_config={
"pod_limits_memory": "6Gi",
"pod_limits_cpu": "1",
}
```
Alternatively, maybe a documentation change to
* https://github.com/apache/airflow/blob/main/docs/apache-airflow/executor/kubernetes.rst
* https://github.com/apache/airflow/blob/main/airflow/example_dags/example_kubernetes_executor_config.py
would make this (IMHO common) task easier
**Use case / motivation**
<!-- What do you want to happen?
Rather than telling us how you might implement this solution, try to take a
step back and describe what you are trying to achieve.
-->
It think the figuring out the `executor_config` > `pod_override` and the `kubernetes.client.models` api just to increase the memory limit for a tasks was kind of hard. I guess a lot of people would benefit from
* exposing a simpler way to achieve it or
* documenting this specific use case in the kubernetes executor documentation
**Are you willing to submit a PR?**
<!--- We accept contributions! -->
**Related Issues**
<!-- Is there currently another issue associated with this? -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16563: executor_config: "limits_memory" option for operators
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16563:
URL: https://github.com/apache/airflow/issues/16563#issuecomment-865043796
Why do you think about creating a factory method?
````
def limits(cpu, memory):
return {
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
limits={
"memory": memory,
"cpu": cpu
}
)
)
]
)
)
````
Usage:
```
task1 = PythonOperator(
task_id="id1",
python_callable=mycallable,
dag=dag1,
executor_config=limits('1', '6Gib')
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #16563: executor_config: "limits_memory" option for operators
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #16563:
URL: https://github.com/apache/airflow/issues/16563#issuecomment-865045209
There is already a `resources` argument to BaseOperator -- if we want some simple way to set those limits we could use that. I think that parameter is currently otherwise unused (it was used for the old Mesos and cgroup executors)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #16563: executor_config: "limits_memory" option for operators
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #16563:
URL: https://github.com/apache/airflow/issues/16563#issuecomment-865045209
There is already a `resources` argument to BaseOperator -- if we want some simple way to set those limits we could use that. I think that parameter is currently otherwise unused (it was used for the old Mesos and cgroup executors)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16563: executor_config: "limits_memory" option for operators
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16563:
URL: https://github.com/apache/airflow/issues/16563#issuecomment-865043796
Why do you think about creating a factory method?
````
def limits(cpu, memory):
return {
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
limits={
"memory": memory,
"cpu": cpu
}
)
)
]
)
)
````
Usage:
```
task1 = PythonOperator(
task_id="id1",
python_callable=mycallable,
dag=dag1,
executor_config=limits('1', '6Gib')
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org