You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "priya369 (via GitHub)" <gi...@apache.org> on 2025/07/29 17:09:31 UTC

[I] Unexpected SIGTERM on Tasks (Airflow 2.10.5 on GKE with KubernetesExecutor and No Resource Constraints) [airflow]

priya369 opened a new issue, #53894:
URL: https://github.com/apache/airflow/issues/53894

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.10.5
   
   ### What happened?
   
   We are running Airflow on a GKE cluster with the KubernetesExecutor. Our deployment has 1200+ DAGs, with 100+ tasks running concurrently at peak. Since the last month, we’ve been observing 20–30 tasks failing daily due to SIGTERM, even though the cluster has no resource pressure (CPU/memory/network usage stays well under 50%).
   We initially suspected it was an issue with the Airflow version (2.9.2), so we upgraded to Airflow 2.10.5 — however, the issue still persists.
   
   [dag_id=PCDM_Editorial_Gold_Top_Content_Append_Daily_run_id=scheduled__2025-07-06T07_15_00+00_00_task_id=GcsToSnowflakegold_top_content_editorial_attempt=1.log](https://github.com/user-attachments/files/21493879/dag_id.PCDM_Editorial_Gold_Top_Content_Append_Daily_run_id.scheduled__2025-07-06T07_15_00%2B00_00_task_id.GcsToSnowflakegold_top_content_editorial_attempt.1.log)
   
   ### What you think should happen instead?
   
   Tasks should continue running unless explicitly cancelled, timeout, or encounter a failure condition. Spontaneous SIGTERM from the Airflow system (without user action or resource pressure) should not happen.
   
   ### How to reproduce
   
   1. airflow config: `---
   ########################################
   ##| Airflow Configs
   ########################################
   
   # User and group of airflow user
   uid: 50000
   gid: 0
   
   securityContext: {}
   airflowHome: /opt/airflow
   defaultAirflowRepository: us-docker.pkg.dev/airflow-003/airflow-docker-image/dataeng
   
   # Images
   images:
     airflow:
       repository: ~
       tag: ~
       pullPolicy: Always
     # To avoid images with user code, you can turn this to 'true' and
     # all the 'run-airflow-migrations' and 'wait-for-airflow-migrations' containers/jobs
     # will use the images from 'defaultAirflowRepository:defaultAirflowTag' values
     # to run and wait for DB migrations .
     useDefaultImageForMigration: true
     # timeout (in seconds) for airflow-migrations to complete
     migrationsWaitTimeout: 300
     statsd:
       repository: quay.io/prometheus/statsd-exporter
       tag: v0.28.0
       pullPolicy: IfNotPresent
     redis:
       repository: redis
       tag: 7-bullseye
       pullPolicy: IfNotPresent
   
   # Select certain nodes for airflow pods.
   nodeSelector: {}
   affinity: {}
   tolerations: []
   topologySpreadConstraints: []
   labels: {}
   
   # Ingress configuration
   ingress:
     # Enable all ingress resources (deprecated - use ingress.web.enabled and ingress.flower.enabled)
     enabled: ~
   
     # Configs for the Ingress of the web Service
     web:
       enabled: true
       annotations:
         kubernetes.io/ingress.class: gce
         kubernetes.io/ingress.allow-http: "false"
         ingress.gcp.kubernetes.io/pre-shared-cert: ott-2024
         networking.gke.io/suppress-firewall-xpn-error: "true"
       path: "/*"
       pathType: "ImplementationSpecific"
       hosts:
         - name: "xxxxxxxxxxxxxxxx"
           tls:
             enabled: false
             secretName: ""
   
       ingressClassName: ""
       precedingPaths: []
       succeedingPaths: []
   
     # Configs for the Ingress of the flower Service
     flower:
       enabled: false
   
   networkPolicies:
     enabled: false
   airflowPodAnnotations:
     gke-gcsfuse/volumes: "true"
   airflowConfigAnnotations: {}
   
   # `airflow_local_settings` file as a string (can be templated).
   airflowLocalSettings: |-
     keepalive_kwargs = {
       "keepalives": 1,
       "keepalives_idle": 30,
       "keepalives_interval": 5,
       "keepalives_count": 5,
     }
   
   rbac:
     create: true
     createSCCRoleBinding: false
   
   executor: "KubernetesExecutor"
   allowPodLaunching: true
   
   # Environment variables for all airflow containers
   env:
     - name: "AIRFLOW_GPL_UNIDECODE"
       value: "yes"
     - name: "AIRFLOW__WEBSERVER__BASE_URL"
       value: "*****************************************"
     - name: "AIRFLOW__CORE__LOAD_EXAMPLES"
       value: "False"
     - name: "AIRFLOW__CORE__LAZY_LOAD_PLUGINS"
       value: "False"
     - name: "AIRFLOW__CORE__PLUGINS_FOLDER"
       value: "/opt/airflow/plugins"
     - name: "AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE"
       value: "True"
     - name: "AIRFLOW__CORE__AIRFLOW_HOME"
       value: "/opt/airflow"
     - name: "AIRFLOW__CORE__DAGS_FOLDER"
       value: "/opt/airflow/dags/repo/airflow-dags"
     - name: "AIRFLOW__CORE__PARALLELISM"
       value: "150"
     - name: "AIRFLOW__CORE__DEFAULT_POOL_TASK_SLOT_COUNT"
       value: "200"
     - name: "AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG"
       value: "60"
     - name: "AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG"
       value: "20"
     - name: "AIRFLOW__CORE__KILLED_TASK_CLEANUP_TIME"
       value: "64000"
     - name: "AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT"
       value: "500"
     - name: "AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT"
       value: "600"
     - name: "AIRFLOW__CORE__ENABLE_XCOM_PICKLING"
       value: "True"
     - name: "AIRFLOW__CORE__TEST_CONNECTION"
       value: "Enabled"
     - name: "AIRFLOW__CORE__MAX_TEMPLATED_FIELD_LENGTH"
       value: "1000000"
     - name: "AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE"
       value: "900"
     - name: "AIRFLOW__DATABASE__SQL_ALCHEMY_MAX_OVERFLOW"
       value: "200"
     - name: "AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE"
       value: "250"
     - name: "AIRFLOW__API__ENABLE_EXPERIMENTAL_API"
       value: "True"
     - name: "AIRFLOW__API__AUTH_BACKEND"
       value: "airflow.api.auth.backend.default"
     - name: "AIRFLOW__WEBSERVER__EXPOSE_CONFIG"
       value: "True"
     - name: "AIRFLOW__WEBSERVER__SHOW_TRIGGER_FORM_IF_NO_PARAMS"
       value: "True"
     - name: "AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT"
       value: "google-cloud-platform://?extra__google_cloud_platform__scope=https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/bigquery"
     - name: "AIRFLOW__LOGGING__LOGGING_LEVEL"
       value: "DEBUG"
     - name: "AIRFLOW__LOGGING__REMOTE_LOGGING"
       value: "True"
     - name: "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER"
       value: "gs://airflow-cluster/airflow/logs"
     - name: "AIRFLOW__KUBERNETES__DAGS_IN_IMAGE"
       value: "False"
     - name: "AIRFLOW__KUBERNETES__NAMESPACE"
       value: "airflow-data-eng-prod"
     - name: "AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST"
       value: "/opt/airflow/dags/repo/airflow-dags"
     - name: "AIRFLOW__KUBERNETES__DAGS_VOLUME_CLAIM"
       value: "room-prod-pvc-pvc"
     - name: "AIRFLOW__KUBERNETES__DAGS_VOLUME_MOUNT_POINT"
       value: "/opt/airflow/dags/repo/airflow-dags"
     - name: "AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME"
       value: "airflow-worker"
     - name: "AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS"
       value: "True"
     - name: "AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS_ON_FAILURE"
       value: "True"
     - name: "AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE"
       value: "50"
     - name: "AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_QUEUED_CHECK_INTERVAL"
       value: "30"
     - name: "AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION"
       value: "False"
     - name: "AIRFLOW__SCHEDULER__SCHEDULER_HEALTH_CHECK_THRESHOLD"
       value: "300"
     - name: "AIRFLOW__SCHEDULER__FILE_PARSING_SORT_MODE"
       value: "modified_time"
     - name: "AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL"
       value: "700"
     - name: "AIRFLOW__SCHEDULER__PARSING_PROCESSES"
       value: "5"
     - name: "AIRFLOW__SCHEDULER__JOB_HEARTBEAT_SEC"
       value: "60"
     - name: "AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC"
       value: "120"
     - name: "AIRFLOW__SCHEDULER__ZOMBIE_DETECTION_INTERVAL"
       value: "100"
     - name: "AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL"
       value: "100"
     - name: "AIRFLOW__WEBSERVER__AUTHENTICATE"
       value: "True"
     - name: "AIRFLOW__WEBSERVER__AUTH_BACKEND"
       value: "airflow.contrib.auth.backends.github_enterprise_auth"
     - name: "AIRFLOW__GITHUB_ENTERPRISE__HOST"
       value: "github.com"
     - name: "AIRFLOW__GITHUB_ENTERPRISE__OAUTH_CALLBACK_ROUTE"
       value: "/oauth-authorized/github"
     
   
   # Volumes for all airflow containers
   volumes: []
   
   # VolumeMounts for all airflow containers
   volumeMounts: []
   
   # Secrets for all airflow containers
   secret: []
   
   # Enables selected built-in secrets that are set via environment variables by default.
   # Those secrets are provided by the Helm Chart secrets by default but in some cases you
   # might want to provide some of those variables with _CMD or _SECRET variable, and you should
   # in this case disable setting of those variables by setting the relevant configuration to false.
   enableBuiltInSecretEnvVars:
     AIRFLOW__CORE__FERNET_KEY: false
     # For Airflow <2.3, backward compatibility; moved to [database] in 2.3
     AIRFLOW__CORE__SQL_ALCHEMY_CONN: true
     AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: true
     AIRFLOW_CONN_AIRFLOW_DB: true
     AIRFLOW__WEBSERVER__SECRET_KEY: false
     AIRFLOW__CELERY__CELERY_RESULT_BACKEND: false
     AIRFLOW__CELERY__RESULT_BACKEND: false
     AIRFLOW__CELERY__BROKER_URL: false
     AIRFLOW__ELASTICSEARCH__HOST: false
     AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST: true
   
   # Extra secrets that will be managed by the chart
   # (You can use them with extraEnv or extraEnvFrom or some of the extraVolumes values).
   # The format for secret data is "key/value" where
   #    * key (can be templated) is the name of the secret that will be created
   #    * value: an object with the standard 'data' or 'stringData' key (or both).
   #          The value associated with those keys must be a string (can be templated)
   extraSecrets: {}
         
   extraEnv: {}
     
   extraEnvFrom: {}
     
   # Airflow database & redis config
   data:
     metadataSecretName: airflow-metadata-connection
     resultBackendSecretName: ~
     brokerUrlSecretName: ~
   
     # Otherwise pass connection values in
     #metadataConnection:
   
     resultBackendConnection: ~
     brokerUrl: ~
   
   # Fernet key settings
   # Note: fernetKey can only be set during install, not upgrade
   fernetKey: ~
   fernetKeySecretName: "***********"
   
   # Flask secret key for Airflow Webserver: `[webserver] secret_key` in airflow.cfg
   webserverSecretKey: ~
   webserverSecretKeySecretName: "*************"
   
   kerberos:
     enabled: false
   
   # Airflow Worker Config
   workers:
     replicas: 0
     revisionHistoryLimit: ~
     command: ~
     args:
       - "bash"
       - "-c"
       # The format below is necessary to get `helm lint` happy
       - |-
         exec \
         airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "celery worker" "worker" }}
   
     livenessProbe:
       enabled: true
       initialDelaySeconds: 10
       timeoutSeconds: 20
       failureThreshold: 5
       periodSeconds: 60
       command: ~
   
     updateStrategy: ~
     strategy:
       rollingUpdate:
         maxSurge: "100%"
         maxUnavailable: "50%"
   
     securityContext: {}
     serviceAccount:
       create: true
       name: airflow-worker
       annotations: {}
   
     keda:
       enabled: false
   
     persistence:
       enabled: false
       size: 100Gi
       storageClassName:
       fixPermissions: false
       annotations: {}
   
     kerberosSidecar:
       enabled: false
       resources: {}
   
     resources:
       limits:
         cpu: 2000m
         memory: 5Gi
       requests:
         cpu: 150m
         memory: 400Mi
   
     terminationGracePeriodSeconds: 600
     safeToEvict: true
     extraContainers: []
     extraInitContainers: []
     extraVolumes:
       {}
     extraVolumeMounts:
       {}
     nodeSelector: {}
     priorityClassName: ~
     affinity: {}
     tolerations: []
     topologySpreadConstraints: []
     hostAliases: []
     podAnnotations: {}
     labels: {}
   
     logGroomerSidecar:
       # Command to use when running the Airflow worker log groomer sidecar (templated).
       command: ~
       # Args to use when running the Airflow worker log groomer sidecar (templated).
       args: ["bash", "/clean-logs"]
       # Number of days to retain logs
       retentionDays: 15
       resources:
         limits:
           cpu: 100m
           memory: 128Mi
         requests:
           cpu: 100m
           memory: 128Mi
   
     waitForMigrations:
       env: []
   
     env: []
   
   scheduler:
     livenessProbe:
       initialDelaySeconds: 10
       timeoutSeconds: 30
       failureThreshold: 5
       periodSeconds: 60
       command: ~
     replicas: 1
     revisionHistoryLimit: ~
     command: ~
     args: ["bash", "-c", "exec airflow scheduler"]
   
     updateStrategy: ~
     strategy: ~
     securityContext: {}
   
     serviceAccount:
       create: true
       name: airflow-scheduler
       annotations: {}
   
     # Scheduler pod disruption budget
     podDisruptionBudget:
       enabled: false
       config:
         maxUnavailable: 1
   
     resources:
       limits:
         cpu: "10"
         memory: 20Gi
       requests:
         cpu: "8"
         memory: 16Gi
   
     # This setting tells kubernetes that its ok to evict
     # when it wants to scale a node down.
     safeToEvict: true
   
     extraContainers: []
     extraInitContainers: []
     extraVolumes:
     nodeSelector: {}
     affinity: {}
     tolerations: []
     topologySpreadConstraints: []
     priorityClassName: ~
     podAnnotations: {}
     labels: {}
   
     logGroomerSidecar:
       enabled: false
       command: ~
       args: ["bash", "/clean-logs"]
       retentionDays: 15
       resources:
         limits:
           cpu: 100m
           memory: 128Mi
         requests:
           cpu: 100m
           memory: 128Mi
   
     waitForMigrations:
       enabled: true
       env: []
   
     env: []
   
   createUserJob:
     command: ~
     args:
       - "bash"
       - "-c"
       # The format below is necessary to get `helm lint` happy
       - |-
         exec \
         airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "users create" "create_user" }} "$@"
       - --
       - "-r"
       - "{{ .Values.webserver.defaultUser.role }}"
       - "-u"
       - "{{ .Values.webserver.defaultUser.username }}"
       - "-e"
       - "{{ .Values.webserver.defaultUser.email }}"
       - "-f"
       - "{{ .Values.webserver.defaultUser.firstName }}"
       - "-l"
       - "{{ .Values.webserver.defaultUser.lastName }}"
       - "-p"
       - "{{ .Values.webserver.defaultUser.password }}"
   
     annotations: {}
     jobAnnotations: {}
     labels: {}
     securityContext: {}
   
     serviceAccount:
       create: true
       name: airflow-createjob
       annotations: {}
   
     extraContainers: []
     extraVolumes: []
     extraVolumeMounts: []
     nodeSelector: {}
     affinity: {}
     tolerations: []
     topologySpreadConstraints: []
     useHelmHooks: false
     applyCustomEnv: false
     env: []
   
     resources:
       limits:
         cpu: 200m
         memory: 512Mi
       requests:
         cpu: 200m
         memory: 512Mi
   
   # Airflow database migration job settings
   migrateDatabaseJob:
     enabled: true
     command: ~
     args:
       - "bash"
       - "-c"
       # The format below is necessary to get `helm lint` happy
       - |-
         exec \
         airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "db upgrade" "upgradedb" }}
   
     annotations: {}
     jobAnnotations: {}
     securityContext: {}
   
     # Create ServiceAccount
     serviceAccount:
       create: true
       name: airflow-migration
       annotations: {}
   
     resources:
       limits:
         cpu: 200m
         memory: 256Mi
       requests:
         cpu: 200m
         memory: 256Mi
   
     # Launch additional containers into database migration job
     extraContainers: []
     extraVolumes: []
     extraVolumeMounts: []
     nodeSelector: {}
     affinity: {}
     tolerations: []
     topologySpreadConstraints: []
     useHelmHooks: false
     applyCustomEnv: false
   
   # Airflow webserver settings
   webserver:
     allowPodLogReading: true
     livenessProbe:
       initialDelaySeconds: 15
       timeoutSeconds: 60
       failureThreshold: 20
       periodSeconds: 5
       scheme: HTTP
   
     readinessProbe:
       initialDelaySeconds: 15
       timeoutSeconds: 30
       failureThreshold: 20
       periodSeconds: 5
       scheme: HTTP
   
     replicas: 1
     revisionHistoryLimit: ~
   
     command: ~
     args: ["bash", "-c", "exec airflow webserver"]
   
     serviceAccount:
       create: true
       name: airflow-web
       annotations: {}
   
     podDisruptionBudget:
       enabled: false
       config:
         maxUnavailable: 1
     strategy: ~
   
     securityContext: {}
     extraNetworkPolicies: []
     networkPolicy:
       ingress:
         from: []
         ports:
           - port: "{{ .Values.ports.airflowUI }}"
   
     resources:
       limits:
         cpu: "7"
         memory: 16Gi
       requests:
         cpu: "5"
         memory: 10Gi
   
     # Create initial user.
     defaultUser:
       enabled: false
   
     # Launch additional containers into webserver.
     extraContainers: []
     extraInitContainers: []
     extraVolumes:
     extraVolumeMounts:
   
     # This string (can be templated) will be mounted into the Airflow Webserver as a custom
     # webserver_config.py. You can bake a webserver_config.py in to your image instead.
     webserverConfig: |
   
     service:
       type: NodePort
       ## service annotations
       annotations: {}
       ports:
         - name: airflow-ui
           port: "{{ .Values.ports.airflowUI }}"
           protocol: TCP
           targetPort: "{{ .Values.ports.airflowUI }}"
       loadBalancerIP: ~
       loadBalancerSourceRanges: []
   
     nodeSelector: {}
     priorityClassName: ~
     affinity: {}
     tolerations: []
     topologySpreadConstraints: []
     podAnnotations: {}
     labels: {}
   
     waitForMigrations:
       enabled: true
       env: []
   
     env: []
   
   # Airflow Triggerer Config
   triggerer:
     enabled: false
   
   # Airflow Dag Processor Config
   dagProcessor:
     enabled: false
   
   # Flower settings
   flower:
     enabled: false
   
   # StatsD settings
   statsd:
     enabled: true
     revisionHistoryLimit: ~
     serviceAccount:
       create: true
       name: ~
       annotations: {}
   
     uid: 65534
     securityContext: {}
   
     # Additional network policies as needed
     extraNetworkPolicies: []
     resources:
       limits:
         cpu: 3000m
         memory: 5Gi
       requests:
         cpu: 2000m
         memory: 3Gi
   
     service:
       extraAnnotations: {}
   
     nodeSelector: {}
     affinity: {}
     tolerations: []
     topologySpreadConstraints: []
     priorityClassName: ~
     extraMappings: []
     overrideMappings: []
     podAnnotations: {}
   
   # PgBouncer settings
   pgbouncer:
     enabled: false
   
   # Configuration for the redis provisioned by the chart
   redis:
     enabled: false
     terminationGracePeriodSeconds: 600
   
     serviceAccount:
       create: false
       name: airflow-redis
       annotations: {}
   
     persistence:
       enabled: false
       size: 1Gi
       storageClassName:
       annotations: {}
   
     resources:
       limits:
         cpu: 300m
         memory: 16Mi
       requests:
         cpu: 100m
         memory: 8Mi
   
     passwordSecretName: ~
     password: ~
     safeToEvict: true
   
     nodeSelector: {}
     affinity: {}
     tolerations: []
     topologySpreadConstraints: []
   
     uid: 0
     securityContext: {}
     podAnnotations: {}
   
   # Auth secret for a private registry
   registry:
     secretName: ~
     connection: {}
   
   # Elasticsearch logging configuration
   elasticsearch:
     enabled: false
   
   # All ports used by chart
   ports:
     flowerUI: 5555
     airflowUI: 8080
     workerLogs: 8793
     redisDB: 6379
     statsdIngest: 9125
     statsdScrape: 9102
     pgbouncer: 6543
     pgbouncerScrape: 9127
   
   # Define any ResourceQuotas for namespace
   quotas: {}
   
   # Define default/max/min values for pods and containers in namespace
   limits: []
   
   # This runs as a CronJob to cleanup old pods.
   cleanup:
     enabled: false
   
   # Configuration for postgresql subchart
   # Not recommended for production
   postgresql:
     enabled: false
   
   # Config settings to go into the mounted airflow.cfg
   #
   # Please note that these values are passed through the `tpl` function, so are
   # all subject to being rendered as go templates. If you need to include a
   # literal `{{` in a value, it must be expressed like this:
   #
   #    a: '{{ "{{ not a template }}" }}'
   #
   # Do not set config containing secrets via plain text values, use Env Var or k8s secret object
   # yamllint disable rule:line-length
   config:
     core:
       dags_folder: '{{ include "airflow_dags" . }}'
       # This is ignored when used with the official Docker image
       load_examples: 'False'
       executor: '{{ .Values.executor }}'
       # For Airflow 1.10, backward compatibility; moved to [logging] in 2.0
       colored_console_log: 'False'
       remote_logging: '{{- ternary "True" "False" .Values.elasticsearch.enabled }}'
     logging:
       remote_logging: '{{- ternary "True" "False" .Values.elasticsearch.enabled }}'
       colored_console_log: 'False'
     metrics:
       statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
       statsd_port: 9125
       statsd_prefix: airflow
       statsd_host: '{{ printf "%s-statsd" .Release.Name }}'
     webserver:
       enable_proxy_fix: 'True'
       # For Airflow 1.10
       rbac: 'True'
       worker_class: 'sync'
       update_fab_perms: 'True'
     celery:
       flower_url_prefix: '{{ .Values.ingress.flower.path }}'
       worker_concurrency: 16
     scheduler:
       standalone_dag_processor: '{{ ternary "True" "False" .Values.dagProcessor.enabled }}'
       # statsd params included for Airflow 1.10 backward compatibility; moved to [metrics] in 2.0
       statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
       statsd_port: 9125
       statsd_prefix: airflow
       statsd_host: '{{ printf "%s-statsd" .Release.Name }}'
       # `run_duration` included for Airflow 1.10 backward compatibility; removed in 2.0.
       run_duration: 41460
     elasticsearch:
       json_format: 'True'
       log_id_template: "{dag_id}_{task_id}_{execution_date}_{try_number}"
     elasticsearch_configs:
       max_retries: 3
       timeout: 30
       retry_timeout: 'True'
     kerberos:
       keytab: '{{ .Values.kerberos.keytabPath }}'
       reinit_frequency: '{{ .Values.kerberos.reinitFrequency }}'
       principal: '{{ .Values.kerberos.principal }}'
       ccache: '{{ .Values.kerberos.ccacheMountPath }}/{{ .Values.kerberos.ccacheFileName }}'
     celery_kubernetes_executor:
       kubernetes_queue: 'kubernetes'
     kubernetes:
       namespace: '{{ .Release.Namespace }}'
       airflow_configmap: '{{ include "airflow_config" . }}'
       airflow_local_settings_configmap: '{{ include "airflow_config" . }}'
       pod_template_file: '{{ include "airflow_pod_template_file" . }}/pod_template_file.yaml'
       worker_container_repository: '{{ .Values.images.airflow.repository | default .Values.defaultAirflowRepository }}'
       worker_container_tag: '{{ .Values.images.airflow.tag | default .Values.defaultAirflowTag }}'
       multi_namespace_mode: '{{ ternary "True" "False" .Values.multiNamespaceMode }}'
     database:
       sql_alchemy_connect_args: airflow_local_settings.keepalive_kwargs
   
   multiNamespaceMode: false
   podTemplate: ~
   
   # Git sync
   dags:
     persistence:
       enabled: true
       size: 5Gi
       storageClassName:
       accessMode: ReadWriteMany
       existingClaim: "xxxxxxxx"
       subPath: ~
     gitSync:
       enabled: false
   
   logs:
     persistence:
       enabled: false
       size: 100Gi
       storageClassName:
       existingClaim:
   
   # CSI Fuse
   
   storageClass:
     name: csi-gcs-fuse-data-eng
   `
   
   ### Operating System
   
   linux
   
   ### Versions of Apache Airflow Providers
   
   2.10.5
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   Airflow Version:
   Originally 2.9.2, upgraded to 2.10.5 (issue persists in both versions)
   
   Executor:
   KubernetesExecutor
   
   Orchestration Platform:
   Google Kubernetes Engine (GKE) – Standard mode (not Autopilot)
   
   Cluster Size & Scaling:
   
   ~20 nodes (n2-highmem-8)
   
   Autoscaling enabled, but cluster consistently runs at <50% CPU/memory usage
   
   Nodes use standard nodepools, not preemptible or spot instances
   
   Airflow Deployment Mode:
   
   Deployed via Helm with custom image (Python 3.10 base)
   
   Webserver, Scheduler, and Triggerer run as separate pods
   
   Using Google Cloud Composer-like architecture, but self-managed
   
   Task Workloads:
   
   ~1200 DAGs, mostly scheduled daily/hourly
   
   ~100–150 concurrent tasks at peak times
   
   Heavy usage of:
   
   DatabricksSubmitRunOperator
   
   SnowflakeOperator
   
   ExternalTaskSensor
   
   ShortCircuitOperator
   
   
   
   ### Anything else?
   
   We have thoroughly verified that this is not caused by:
   - Resource exhaustion (CPU, memory, or disk)
   - Pod evictions or node preemptions
   - Task timeouts or DAG-level retries
   - Manual task terminations
   - Implemented dag timeout and execution timeout also created separate pool for databricks and snowflake task
   
   We suspect the issue may stem from one of the following:
   -Internal Airflow lifecycle logic (e.g., zombie detection, pod orphan cleanup)
   - K8sExecutor edge cases where the scheduler or triggerer may initiate SIGTERM without resource-based justification
   - GCSFuse or sidecar instability possibly impacting task pod lifecycle (though main containers are healthy)
   - We are open to instrumenting Airflow internals with logging or tracing if the core team can suggest areas to probe (e.g: executor heartbeat, cleanup routines, orphan detection)
   
   We would also be happy to help test a patch or proposed fix in our production-like test cluster if needed.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Unexpected SIGTERM on Tasks (Airflow 2.10.5 on GKE with KubernetesExecutor and No Resource Constraints) [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #53894:
URL: https://github.com/apache/airflow/issues/53894#issuecomment-3133369635

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Unexpected SIGTERM on Tasks (Airflow 2.10.5 on GKE with KubernetesExecutor and No Resource Constraints) [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #53894: Unexpected SIGTERM on Tasks (Airflow 2.10.5 on GKE with KubernetesExecutor and No Resource Constraints)
URL: https://github.com/apache/airflow/issues/53894


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Unexpected SIGTERM on Tasks (Airflow 2.10.5 on GKE with KubernetesExecutor and No Resource Constraints) [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #53894:
URL: https://github.com/apache/airflow/issues/53894#issuecomment-3139162455

   I think your best bet is to attempt to upgrade to Airflow 3. This is issue might take forever to diagnose because there is absolutely no clue on why it happens. The whole mechanism of heartbeating, supervising running tasks, etc. that could be causeing it had been completely rewritten there and I think the fastest way to get this problem solved is to simply switch to Airflow 3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org