You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Jon Davies (JIRA)" <ji...@apache.org> on 2018/08/24 16:39:00 UTC
[jira] [Created] (AIRFLOW-2955) Kubernetes pod operator: Unable to
set requests/limits on task pods
Jon Davies created AIRFLOW-2955:
-----------------------------------
Summary: Kubernetes pod operator: Unable to set requests/limits on task pods
Key: AIRFLOW-2955
URL: https://issues.apache.org/jira/browse/AIRFLOW-2955
Project: Apache Airflow
Issue Type: Bug
Reporter: Jon Davies
When I try and set a resource limit/request on a DAG task with the KubernetesPodOperator, I get:
{code:java}
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task Traceback (most recent call last):
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/bin/airflow", line 32, in <module>
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task args.func(args)
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task return f(*args, **kwargs)
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 498, in run
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task _run(args, dag, ti)
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 402, in _run
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task pool=args.pool,
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task return func(*args, **kwargs)
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/models.py", line 1633, in _run_raw_task
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task result = task_copy.execute(context=context)
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 115, in execute
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task get_logs=self.get_logs)
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 71, in run_pod
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task resp = self.run_pod_async(pod)
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 52, in run_pod_async
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task req = self.kube_req_factory.create(pod)
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py", line 56, in create
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task self.extract_resources(pod, req)
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task File "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/kubernetes_request_factory.py", line 160, in extract_resources
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task if not pod.resources or pod.resources.is_empty_resource_request():
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task AttributeError: 'dict' object has no attribute 'is_empty_resource_request'
{code}
...setting https://github.com/apache/incubator-airflow/blob/fc10f7e0a04145a0b2f31f8d0990bbe900b4e8a2/airflow/example_dags/example_kubernetes_executor.py#L66 works, however that only adjusts the metadata for the worker pod.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)