You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2019/06/18 09:35:03 UTC
[jira] [Commented] (AIRFLOW-2955) Kubernetes pod operator: Unable
to set requests/limits on task pods
[ https://issues.apache.org/jira/browse/AIRFLOW-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866403#comment-16866403 ]
ASF subversion and git services commented on AIRFLOW-2955:
----------------------------------------------------------
Commit 9c96fd9d5d39c60111f946c04b6ceabf41476405 in airflow's branch refs/heads/v1-10-stable from Summit Suen
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9c96fd9 ]
[AIRFLOW-2955] Fix kubernetes pod operator to set requests and limits on task pods (#4551)
* [AIRFLOW-2955] Fix kubernetes pod operator to set requests and limits on task pods.
* [AIRFLOW-2955] Remove bare except to follow flake8.
* [AIRFLOW-2955] Remove unused library.
* [AIRFLOW-2955] Fix kubernetes pod operator to set requests and limits on task pods.
* [AIRFLOW-2955] Remove bare except to follow flake8.
* [AIRFLOW-2955] Remove unused library.
* Resolved conflicts.
* [AIRFLOW-2955] Fix kubernetes pod operator to set requests and limits on task pods.
* [AIRFLOW-2955] Remove bare except to follow flake8.
* [AIRFLOW-2955] Remove unused library.
* Resolved conflicts.
* Resolve conflicts.
* [AIRFLOW-2955] Remove bare except to follow flake8.
* [AIRFLOW-2955] Remove unused library.
* [AIRFLOW-2955] clear up commits.
* Resolve nits form @galuszkak and @dimberman.
(cherry picked from commit 82c7b450e3d360dca3feae556503460f1144266c)
> Kubernetes pod operator: Unable to set requests/limits on task pods
> -------------------------------------------------------------------
>
> Key: AIRFLOW-2955
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2955
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Jon Davies
> Assignee: Summit Suen
> Priority: Major
> Fix For: 1.10.4
>
>
> When I try and set a resource limit/request on a DAG task with the KubernetesPodOperator as follows:
> {code:java}
> resources={"limit_cpu": 1, "request_cpu": 1},
> {code}
> ...I get:
> {code:java}
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task Traceback (most recent call last):
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/bin/airflow", line 32, in <module>
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task args.func(args)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task return f(*args, **kwargs)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 498, in run
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task _run(args, dag, ti)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 402, in _run
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task pool=args.pool,
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task return func(*args, **kwargs)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/models.py", line 1633, in _run_raw_task
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task result = task_copy.execute(context=context)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 115, in execute
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task get_logs=self.get_logs)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 71, in run_pod
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task resp = self.run_pod_async(pod)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 52, in run_pod_async
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task req = self.kube_req_factory.create(pod)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py", line 56, in create
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task self.extract_resources(pod, req)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/kubernetes_request_factory.py", line 160, in extract_resources
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task if not pod.resources or pod.resources.is_empty_resource_request():
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task AttributeError: 'dict' object has no attribute 'is_empty_resource_request'
> {code}
> ...setting https://github.com/apache/incubator-airflow/blob/fc10f7e0a04145a0b2f31f8d0990bbe900b4e8a2/airflow/example_dags/example_kubernetes_executor.py#L66 works, however that only adjusts the metadata for the worker pod and not the pod ultimately used for the task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)