You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/09 17:15:06 UTC
[GitHub] [airflow] hedrickw opened a new issue #12960: Attempt to use Pod Template File with KubernetesExecutor version 1.10.13 Throws AttributeError: 'V1EnvFromSource' object has no attribute 'name'
hedrickw opened a new issue #12960:
URL: https://github.com/apache/airflow/issues/12960
**Apache Airflow version**:
1.10.13
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
1.18.8"
**Environment**:
Ubuntu 16.04
- **Install tools**:
Helm Chart
**What happened**:
Attempt to use Pod Template File with KubernetesExecutor version 1.10.13 Throws AttributeError: 'V1EnvFromSource' object has no attribute 'name'
> Template file being sent to pod
```
kind: Pod
apiVersion: v1
metadata:
name: mando_yaml
namespace: default
labels:
mylabel: mando_pod
spec:
containers:
- name: mando
image: example
env:
- name: AIRFLOW__CORE_DAGS_FOLDER
value: /src/example/apache
- name: AIRFLOW_HOME
value: /src/example/apache
- name: AIRFLOW__CORE__EXECUTOR
value: LocalExecutor
- name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
value: postgresql+psycopg2://postgres:airflow@example-postgresql:5432/airflow
envFrom:
- configMapRef:
name: example-env
imagePullPolicy: Always
ports: []
volumeMounts:
- mountPath: /opt/airflow/logs
name: airflow-logs
resources:
limits:
memory: "500Mi"
requests:
memory: "500Mi"
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
hostNetwork: false
imagePullSecrets: []
initContainers: []
nodeSelector: {}
restartPolicy: Never
securityContext:
runAsUser: 0
serviceAccountName: example
volumes:
- emptyDir: {}
name: airflow-logs
```
> Error
```
['airflow', 'run', 'mandalore_pod_template_limits_v2', 'mandalore_saying', '2020-12-09T16:07:19.537177+00:00', '--local', '--pool', 'default_pool', '-sd', '/src/examples/apache/dags/mando_pod_template_v2.py'], {'api_version': 'v1',
'kind': 'Pod',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': None,
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': {'mylabel': 'mando_pod'},
'managed_fields': None,
'name': 'mando_yaml-a2520ce7604c40b2a84338cd639b8815',
'namespace': 'default',
'owner_references': None,
'resource_version': None,
'self_link': None,
'uid': None},
'spec': {'active_deadline_seconds': None,
'affinity': None,
'automount_service_account_token': None,
'containers': [{'args': None,
'command': None,
'env': [{'name': 'AIRFLOW__CORE_DAGS_FOLDER',
'value': '/src/example/apache',
'value_from': None},
{'name': 'AIRFLOW_HOME',
'value': '/src/example/apache',
'value_from': None},
{'name': 'AIRFLOW__CORE__EXECUTOR',
'value': 'LocalExecutor',
'value_from': None},
{'name': 'AIRFLOW__CORE__SQL_ALCHEMY_CONN',
'value': 'postgresql+psycopg2://postgres:airflow@example-postgresql:5432/airflow',
'value_from': NonE
'env_from': [{'config_map_ref': {'name': 'example-env',
'optional': None},
'prefix': None,
'secret_ref': None}],
'image': 'example',
'image_pull_policy': 'Always',
'lifecycle': None,
'liveness_probe': None,
'name': 'mando',
'ports': [],
'readiness_probe': None,
'resources': {'limits': {'memory': '500Mi'},
'requests': {'memory': '500Mi'}},
'security_context': None,
'stdin': None,
'stdin_once': None,
'termination_message_path': '/dev/termination-log',
'termination_message_policy': 'File',
'tty': None,
'volume_devices': None,
'volume_mounts': [{'mount_path': '/opt/airflow/logs',
'mount_propagation': None,
'name': 'airflow-logs',
'read_only': None,
'sub_path': None,
'sub_path_expr': None}],
'working_dir': None}],
'dns_config': None,
'dns_policy': None,
'enable_service_links': None,
'host_aliases': None,
'host_ipc': None,
'host_network': False,
'host_pid': None,
'hostname': None,
'image_pull_secrets': [],
'init_containers': [],
'node_name': None,
'node_selector': {},
'preemption_policy': None,
'priority': None,
'priority_class_name': None,
'readiness_gates': None,
'restart_policy': 'Never',
'runtime_class_name': None,
'scheduler_name': None,
'security_context': {'fs_group': None,
'run_as_group': None,
'run_as_non_root': None,
'run_as_user': 0,
'se_linux_options': None,
'supplemental_groups': None,
'sysctls': None,
'windows_options': None},
'service_account': None,
'service_account_name': 'odin',
'share_process_namespace': None,
'subdomain': None,
'termination_grace_period_seconds': None,
'tolerations': None,
'volumes': [{'aws_elastic_block_store': None,
'azure_disk': None,
'azure_file': None,
'cephfs': None,
'cinder': None,
'config_map': None,
'csi': None,
'downward_api': None,
'empty_dir': {'medium': None, 'size_limit': None},
'fc': None,
'flex_volume': None,
'flocker': None,
'gce_persistent_disk': None,
'git_repo': None,
'glusterfs': None,
'host_path': None,
'iscsi': None,
'name': 'airflow-logs',
'nfs': None,
'persistent_volume_claim': None,
'photon_persistent_disk': None,
'portworx_volume': None,
'projected': None,
'quobyte': None,
'rbd': None,
'scale_io': None,
'secret': None,
'storageos': None,
'vsphere_volume': None}]},
'status': None})
[2020-12-09 16:43:20,697] {scheduler_job.py:1384} ERROR - Exception when executing execute_helper
Traceback (most recent call last):
File "/opt/app-root/lib64/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1382, in _execute
self._execute_helper()
File "/opt/app-root/lib64/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1453, in _execute_helper
if not self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag):
File "/opt/app-root/lib64/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1515, in _validate_and_run_task_instances
self.executor.heartbeat()
File "/opt/app-root/lib64/python3.8/site-packages/airflow/executors/base_executor.py", line 134, in heartbeat
self.sync()
File "/opt/app-root/lib64/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 832, in sync
self.kube_scheduler.run_next(task)
File "/opt/app-root/lib64/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 436, in run_next
pod = PodGenerator.construct_pod(
File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 564, in construct_pod
return reduce(PodGenerator.reconcile_pods, pod_list)
File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 429, in reconcile_pods
client_pod_cp.spec = PodGenerator.reconcile_specs(base_pod.spec, client_pod_cp.spec)
File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 475, in reconcile_specs
client_spec.containers = PodGenerator.reconcile_containers(
File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 509, in reconcile_containers
client_container = extend_object_field(base_container, client_container, 'env_from')
File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 647, in extend_object_field
base_obj_set = _get_dict_from_list(base_obj_field, field_to_merge)
File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 676, in _get_dict_from_list
result[getattr(obj, field_to_merge)] = obj
AttributeError: 'V1EnvFromSource' object has no attribute 'name'
```
```
from datetime import timedelta
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.utils.dates import days_ago
from kubernetes.client import models as k8s
import os
def mando_saying():
print("This is the Way")
print(os.getenv("TEST"))
default_args = {
"owner": "airflow",
"depends_on_past": False,
"start_date": days_ago(2),
"email": ["airflow@example.com"],
"email_on_failure": False,
"email_on_retry": False,
"retries": 0,
"retry_delay": timedelta(minutes=5),
}
dag = DAG("mandalore_pod_template_limits_v2", default_args=default_args, description="Test Script", schedule_interval=None)
this_is_the_way = PythonOperator(
task_id="mandalore_saying",
python_callable=mando_saying,
dag=dag,
depends_on_past=False,
executor_config={"KubernetesExecutor": {"pod_template_file": "/src/explorer2/apache/pod_templates/mando.yaml",}},
)
```
I tried doing it this way because when i change the executor_config to
`executor_config={"pod_template_file": "/src/explorer2/apache/pod_templates/mando.yaml",},`
The Scheduler is not reading the change and just launches the pod with all the defaults :/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hedrickw closed issue #12960: Attempt to use Pod Template File with KubernetesExecutor version 1.10.13 Throws AttributeError: 'V1EnvFromSource' object has no attribute 'name'
Posted by GitBox <gi...@apache.org>.
hedrickw closed issue #12960:
URL: https://github.com/apache/airflow/issues/12960
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #12960: Attempt to use Pod Template File with KubernetesExecutor version 1.10.13 Throws AttributeError: 'V1EnvFromSource' object has no attribute 'name'
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12960:
URL: https://github.com/apache/airflow/issues/12960#issuecomment-741915570
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org