You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/11 18:06:02 UTC
[GitHub] [airflow] cris-b opened a new issue #13023: AttributeError when merging env_from in pod_generator
cris-b opened a new issue #13023:
URL: https://github.com/apache/airflow/issues/13023
**Apache Airflow version**: 1.10.14
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.19.3
**Environment**:
- **Cloud provider or hardware configuration**: Azure
- **OS** (e.g. from /etc/os-release): RHEL8
**What happened**:
In my executor_config I had an env_from. I also wanted to pass some env as secrets but cant do that directly in executor_config so had to use the AIRFLOW__KUBERNETES__ENV_FROM_SECRET_REF variable.
This led to a crash when trying to create the pod:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1382, in _execute
self._execute_helper()
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1453, in _execute_helper
if not self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag):
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1515, in _validate_and_run_task_instances
self.executor.heartbeat()
File "/usr/local/lib/python3.7/site-packages/airflow/executors/base_executor.py", line 134, in heartbeat
self.sync()
File "/usr/local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 832, in sync
self.kube_scheduler.run_next(task)
File "/usr/local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 447, in run_next
base_worker_pod=self.worker_configuration_pod
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 564, in construct_pod
return reduce(PodGenerator.reconcile_pods, pod_list)
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 429, in reconcile_pods
client_pod_cp.spec = PodGenerator.reconcile_specs(base_pod.spec, client_pod_cp.spec)
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 476, in reconcile_specs
base_spec.containers, client_spec.containers
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 509, in reconcile_containers
client_container = extend_object_field(base_container, client_container, 'env_from')
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 647, in extend_object_field
base_obj_set = _get_dict_from_list(base_obj_field, field_to_merge)
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 676, in _get_dict_from_list
result[getattr(obj, field_to_merge)] = obj
AttributeError: 'V1EnvFromSource' object has no attribute 'name'
**What you expected to happen**:
Pod created successfully
**How to reproduce it**:
Define a pod in executor_config (or pod_template) and additionally use AIRFLOW__KUBERNETES__ENV_FROM_SECRET_REF or AIRFLOW__KUBERNETES__ENV_FROM_CONFIGMAP_REF
As long as both the base config and the template contain envFrom/env_from.
**Anything else we need to know**:
This is caused by PodGenerator.extend_object_field trying to merge on 'name' but V1EnvFromSource don't have those, they either specify config_map_ref or secret_ref.
<!--
How often does this problem occur? Once? Every time etc?
Every time.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #13023: AttributeError when merging env_from in pod_generator
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #13023:
URL: https://github.com/apache/airflow/issues/13023#issuecomment-773582329
@cris-b would you be interested in merging your fix? I'm glad to help you get setup with breeze/review your PR if you have cycles :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cris-b commented on issue #13023: AttributeError when merging env_from in pod_generator
Posted by GitBox <gi...@apache.org>.
cris-b commented on issue #13023:
URL: https://github.com/apache/airflow/issues/13023#issuecomment-743734853
Broken by [ #10366 ]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13023: AttributeError when merging env_from in pod_generator
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13023:
URL: https://github.com/apache/airflow/issues/13023#issuecomment-794388500
closed by https://github.com/apache/airflow/pull/14090
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #13023: AttributeError when merging env_from in pod_generator
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #13023:
URL: https://github.com/apache/airflow/issues/13023
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #13023: AttributeError when merging env_from in pod_generator
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #13023:
URL: https://github.com/apache/airflow/issues/13023#issuecomment-773582329
@cris-b would you be interested in merging your fix? I'm glad to help you get setup with breeze/review your PR if you have cycles :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org