You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "priya369 (via GitHub)" <gi...@apache.org> on 2025/07/29 17:09:31 UTC
[I] Unexpected SIGTERM on Tasks (Airflow 2.10.5 on GKE with KubernetesExecutor and No Resource Constraints) [airflow]
priya369 opened a new issue, #53894:
URL: https://github.com/apache/airflow/issues/53894
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.10.5
### What happened?
We are running Airflow on a GKE cluster with the KubernetesExecutor. Our deployment has 1200+ DAGs, with 100+ tasks running concurrently at peak. Since the last month, we’ve been observing 20–30 tasks failing daily due to SIGTERM, even though the cluster has no resource pressure (CPU/memory/network usage stays well under 50%).
We initially suspected it was an issue with the Airflow version (2.9.2), so we upgraded to Airflow 2.10.5 — however, the issue still persists.
[dag_id=PCDM_Editorial_Gold_Top_Content_Append_Daily_run_id=scheduled__2025-07-06T07_15_00+00_00_task_id=GcsToSnowflakegold_top_content_editorial_attempt=1.log](https://github.com/user-attachments/files/21493879/dag_id.PCDM_Editorial_Gold_Top_Content_Append_Daily_run_id.scheduled__2025-07-06T07_15_00%2B00_00_task_id.GcsToSnowflakegold_top_content_editorial_attempt.1.log)
### What you think should happen instead?
Tasks should continue running unless explicitly cancelled, timeout, or encounter a failure condition. Spontaneous SIGTERM from the Airflow system (without user action or resource pressure) should not happen.
### How to reproduce
1. airflow config: `---
########################################
##| Airflow Configs
########################################
# User and group of airflow user
uid: 50000
gid: 0
securityContext: {}
airflowHome: /opt/airflow
defaultAirflowRepository: us-docker.pkg.dev/airflow-003/airflow-docker-image/dataeng
# Images
images:
airflow:
repository: ~
tag: ~
pullPolicy: Always
# To avoid images with user code, you can turn this to 'true' and
# all the 'run-airflow-migrations' and 'wait-for-airflow-migrations' containers/jobs
# will use the images from 'defaultAirflowRepository:defaultAirflowTag' values
# to run and wait for DB migrations .
useDefaultImageForMigration: true
# timeout (in seconds) for airflow-migrations to complete
migrationsWaitTimeout: 300
statsd:
repository: quay.io/prometheus/statsd-exporter
tag: v0.28.0
pullPolicy: IfNotPresent
redis:
repository: redis
tag: 7-bullseye
pullPolicy: IfNotPresent
# Select certain nodes for airflow pods.
nodeSelector: {}
affinity: {}
tolerations: []
topologySpreadConstraints: []
labels: {}
# Ingress configuration
ingress:
# Enable all ingress resources (deprecated - use ingress.web.enabled and ingress.flower.enabled)
enabled: ~
# Configs for the Ingress of the web Service
web:
enabled: true
annotations:
kubernetes.io/ingress.class: gce
kubernetes.io/ingress.allow-http: "false"
ingress.gcp.kubernetes.io/pre-shared-cert: ott-2024
networking.gke.io/suppress-firewall-xpn-error: "true"
path: "/*"
pathType: "ImplementationSpecific"
hosts:
- name: "xxxxxxxxxxxxxxxx"
tls:
enabled: false
secretName: ""
ingressClassName: ""
precedingPaths: []
succeedingPaths: []
# Configs for the Ingress of the flower Service
flower:
enabled: false
networkPolicies:
enabled: false
airflowPodAnnotations:
gke-gcsfuse/volumes: "true"
airflowConfigAnnotations: {}
# `airflow_local_settings` file as a string (can be templated).
airflowLocalSettings: |-
keepalive_kwargs = {
"keepalives": 1,
"keepalives_idle": 30,
"keepalives_interval": 5,
"keepalives_count": 5,
}
rbac:
create: true
createSCCRoleBinding: false
executor: "KubernetesExecutor"
allowPodLaunching: true
# Environment variables for all airflow containers
env:
- name: "AIRFLOW_GPL_UNIDECODE"
value: "yes"
- name: "AIRFLOW__WEBSERVER__BASE_URL"
value: "*****************************************"
- name: "AIRFLOW__CORE__LOAD_EXAMPLES"
value: "False"
- name: "AIRFLOW__CORE__LAZY_LOAD_PLUGINS"
value: "False"
- name: "AIRFLOW__CORE__PLUGINS_FOLDER"
value: "/opt/airflow/plugins"
- name: "AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE"
value: "True"
- name: "AIRFLOW__CORE__AIRFLOW_HOME"
value: "/opt/airflow"
- name: "AIRFLOW__CORE__DAGS_FOLDER"
value: "/opt/airflow/dags/repo/airflow-dags"
- name: "AIRFLOW__CORE__PARALLELISM"
value: "150"
- name: "AIRFLOW__CORE__DEFAULT_POOL_TASK_SLOT_COUNT"
value: "200"
- name: "AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG"
value: "60"
- name: "AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG"
value: "20"
- name: "AIRFLOW__CORE__KILLED_TASK_CLEANUP_TIME"
value: "64000"
- name: "AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT"
value: "500"
- name: "AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT"
value: "600"
- name: "AIRFLOW__CORE__ENABLE_XCOM_PICKLING"
value: "True"
- name: "AIRFLOW__CORE__TEST_CONNECTION"
value: "Enabled"
- name: "AIRFLOW__CORE__MAX_TEMPLATED_FIELD_LENGTH"
value: "1000000"
- name: "AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE"
value: "900"
- name: "AIRFLOW__DATABASE__SQL_ALCHEMY_MAX_OVERFLOW"
value: "200"
- name: "AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE"
value: "250"
- name: "AIRFLOW__API__ENABLE_EXPERIMENTAL_API"
value: "True"
- name: "AIRFLOW__API__AUTH_BACKEND"
value: "airflow.api.auth.backend.default"
- name: "AIRFLOW__WEBSERVER__EXPOSE_CONFIG"
value: "True"
- name: "AIRFLOW__WEBSERVER__SHOW_TRIGGER_FORM_IF_NO_PARAMS"
value: "True"
- name: "AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT"
value: "google-cloud-platform://?extra__google_cloud_platform__scope=https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/bigquery"
- name: "AIRFLOW__LOGGING__LOGGING_LEVEL"
value: "DEBUG"
- name: "AIRFLOW__LOGGING__REMOTE_LOGGING"
value: "True"
- name: "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER"
value: "gs://airflow-cluster/airflow/logs"
- name: "AIRFLOW__KUBERNETES__DAGS_IN_IMAGE"
value: "False"
- name: "AIRFLOW__KUBERNETES__NAMESPACE"
value: "airflow-data-eng-prod"
- name: "AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST"
value: "/opt/airflow/dags/repo/airflow-dags"
- name: "AIRFLOW__KUBERNETES__DAGS_VOLUME_CLAIM"
value: "room-prod-pvc-pvc"
- name: "AIRFLOW__KUBERNETES__DAGS_VOLUME_MOUNT_POINT"
value: "/opt/airflow/dags/repo/airflow-dags"
- name: "AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME"
value: "airflow-worker"
- name: "AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS"
value: "True"
- name: "AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS_ON_FAILURE"
value: "True"
- name: "AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE"
value: "50"
- name: "AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_QUEUED_CHECK_INTERVAL"
value: "30"
- name: "AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION"
value: "False"
- name: "AIRFLOW__SCHEDULER__SCHEDULER_HEALTH_CHECK_THRESHOLD"
value: "300"
- name: "AIRFLOW__SCHEDULER__FILE_PARSING_SORT_MODE"
value: "modified_time"
- name: "AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL"
value: "700"
- name: "AIRFLOW__SCHEDULER__PARSING_PROCESSES"
value: "5"
- name: "AIRFLOW__SCHEDULER__JOB_HEARTBEAT_SEC"
value: "60"
- name: "AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC"
value: "120"
- name: "AIRFLOW__SCHEDULER__ZOMBIE_DETECTION_INTERVAL"
value: "100"
- name: "AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL"
value: "100"
- name: "AIRFLOW__WEBSERVER__AUTHENTICATE"
value: "True"
- name: "AIRFLOW__WEBSERVER__AUTH_BACKEND"
value: "airflow.contrib.auth.backends.github_enterprise_auth"
- name: "AIRFLOW__GITHUB_ENTERPRISE__HOST"
value: "github.com"
- name: "AIRFLOW__GITHUB_ENTERPRISE__OAUTH_CALLBACK_ROUTE"
value: "/oauth-authorized/github"
# Volumes for all airflow containers
volumes: []
# VolumeMounts for all airflow containers
volumeMounts: []
# Secrets for all airflow containers
secret: []
# Enables selected built-in secrets that are set via environment variables by default.
# Those secrets are provided by the Helm Chart secrets by default but in some cases you
# might want to provide some of those variables with _CMD or _SECRET variable, and you should
# in this case disable setting of those variables by setting the relevant configuration to false.
enableBuiltInSecretEnvVars:
AIRFLOW__CORE__FERNET_KEY: false
# For Airflow <2.3, backward compatibility; moved to [database] in 2.3
AIRFLOW__CORE__SQL_ALCHEMY_CONN: true
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: true
AIRFLOW_CONN_AIRFLOW_DB: true
AIRFLOW__WEBSERVER__SECRET_KEY: false
AIRFLOW__CELERY__CELERY_RESULT_BACKEND: false
AIRFLOW__CELERY__RESULT_BACKEND: false
AIRFLOW__CELERY__BROKER_URL: false
AIRFLOW__ELASTICSEARCH__HOST: false
AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST: true
# Extra secrets that will be managed by the chart
# (You can use them with extraEnv or extraEnvFrom or some of the extraVolumes values).
# The format for secret data is "key/value" where
# * key (can be templated) is the name of the secret that will be created
# * value: an object with the standard 'data' or 'stringData' key (or both).
# The value associated with those keys must be a string (can be templated)
extraSecrets: {}
extraEnv: {}
extraEnvFrom: {}
# Airflow database & redis config
data:
metadataSecretName: airflow-metadata-connection
resultBackendSecretName: ~
brokerUrlSecretName: ~
# Otherwise pass connection values in
#metadataConnection:
resultBackendConnection: ~
brokerUrl: ~
# Fernet key settings
# Note: fernetKey can only be set during install, not upgrade
fernetKey: ~
fernetKeySecretName: "***********"
# Flask secret key for Airflow Webserver: `[webserver] secret_key` in airflow.cfg
webserverSecretKey: ~
webserverSecretKeySecretName: "*************"
kerberos:
enabled: false
# Airflow Worker Config
workers:
replicas: 0
revisionHistoryLimit: ~
command: ~
args:
- "bash"
- "-c"
# The format below is necessary to get `helm lint` happy
- |-
exec \
airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "celery worker" "worker" }}
livenessProbe:
enabled: true
initialDelaySeconds: 10
timeoutSeconds: 20
failureThreshold: 5
periodSeconds: 60
command: ~
updateStrategy: ~
strategy:
rollingUpdate:
maxSurge: "100%"
maxUnavailable: "50%"
securityContext: {}
serviceAccount:
create: true
name: airflow-worker
annotations: {}
keda:
enabled: false
persistence:
enabled: false
size: 100Gi
storageClassName:
fixPermissions: false
annotations: {}
kerberosSidecar:
enabled: false
resources: {}
resources:
limits:
cpu: 2000m
memory: 5Gi
requests:
cpu: 150m
memory: 400Mi
terminationGracePeriodSeconds: 600
safeToEvict: true
extraContainers: []
extraInitContainers: []
extraVolumes:
{}
extraVolumeMounts:
{}
nodeSelector: {}
priorityClassName: ~
affinity: {}
tolerations: []
topologySpreadConstraints: []
hostAliases: []
podAnnotations: {}
labels: {}
logGroomerSidecar:
# Command to use when running the Airflow worker log groomer sidecar (templated).
command: ~
# Args to use when running the Airflow worker log groomer sidecar (templated).
args: ["bash", "/clean-logs"]
# Number of days to retain logs
retentionDays: 15
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
waitForMigrations:
env: []
env: []
scheduler:
livenessProbe:
initialDelaySeconds: 10
timeoutSeconds: 30
failureThreshold: 5
periodSeconds: 60
command: ~
replicas: 1
revisionHistoryLimit: ~
command: ~
args: ["bash", "-c", "exec airflow scheduler"]
updateStrategy: ~
strategy: ~
securityContext: {}
serviceAccount:
create: true
name: airflow-scheduler
annotations: {}
# Scheduler pod disruption budget
podDisruptionBudget:
enabled: false
config:
maxUnavailable: 1
resources:
limits:
cpu: "10"
memory: 20Gi
requests:
cpu: "8"
memory: 16Gi
# This setting tells kubernetes that its ok to evict
# when it wants to scale a node down.
safeToEvict: true
extraContainers: []
extraInitContainers: []
extraVolumes:
nodeSelector: {}
affinity: {}
tolerations: []
topologySpreadConstraints: []
priorityClassName: ~
podAnnotations: {}
labels: {}
logGroomerSidecar:
enabled: false
command: ~
args: ["bash", "/clean-logs"]
retentionDays: 15
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
waitForMigrations:
enabled: true
env: []
env: []
createUserJob:
command: ~
args:
- "bash"
- "-c"
# The format below is necessary to get `helm lint` happy
- |-
exec \
airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "users create" "create_user" }} "$@"
- --
- "-r"
- "{{ .Values.webserver.defaultUser.role }}"
- "-u"
- "{{ .Values.webserver.defaultUser.username }}"
- "-e"
- "{{ .Values.webserver.defaultUser.email }}"
- "-f"
- "{{ .Values.webserver.defaultUser.firstName }}"
- "-l"
- "{{ .Values.webserver.defaultUser.lastName }}"
- "-p"
- "{{ .Values.webserver.defaultUser.password }}"
annotations: {}
jobAnnotations: {}
labels: {}
securityContext: {}
serviceAccount:
create: true
name: airflow-createjob
annotations: {}
extraContainers: []
extraVolumes: []
extraVolumeMounts: []
nodeSelector: {}
affinity: {}
tolerations: []
topologySpreadConstraints: []
useHelmHooks: false
applyCustomEnv: false
env: []
resources:
limits:
cpu: 200m
memory: 512Mi
requests:
cpu: 200m
memory: 512Mi
# Airflow database migration job settings
migrateDatabaseJob:
enabled: true
command: ~
args:
- "bash"
- "-c"
# The format below is necessary to get `helm lint` happy
- |-
exec \
airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "db upgrade" "upgradedb" }}
annotations: {}
jobAnnotations: {}
securityContext: {}
# Create ServiceAccount
serviceAccount:
create: true
name: airflow-migration
annotations: {}
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 200m
memory: 256Mi
# Launch additional containers into database migration job
extraContainers: []
extraVolumes: []
extraVolumeMounts: []
nodeSelector: {}
affinity: {}
tolerations: []
topologySpreadConstraints: []
useHelmHooks: false
applyCustomEnv: false
# Airflow webserver settings
webserver:
allowPodLogReading: true
livenessProbe:
initialDelaySeconds: 15
timeoutSeconds: 60
failureThreshold: 20
periodSeconds: 5
scheme: HTTP
readinessProbe:
initialDelaySeconds: 15
timeoutSeconds: 30
failureThreshold: 20
periodSeconds: 5
scheme: HTTP
replicas: 1
revisionHistoryLimit: ~
command: ~
args: ["bash", "-c", "exec airflow webserver"]
serviceAccount:
create: true
name: airflow-web
annotations: {}
podDisruptionBudget:
enabled: false
config:
maxUnavailable: 1
strategy: ~
securityContext: {}
extraNetworkPolicies: []
networkPolicy:
ingress:
from: []
ports:
- port: "{{ .Values.ports.airflowUI }}"
resources:
limits:
cpu: "7"
memory: 16Gi
requests:
cpu: "5"
memory: 10Gi
# Create initial user.
defaultUser:
enabled: false
# Launch additional containers into webserver.
extraContainers: []
extraInitContainers: []
extraVolumes:
extraVolumeMounts:
# This string (can be templated) will be mounted into the Airflow Webserver as a custom
# webserver_config.py. You can bake a webserver_config.py in to your image instead.
webserverConfig: |
service:
type: NodePort
## service annotations
annotations: {}
ports:
- name: airflow-ui
port: "{{ .Values.ports.airflowUI }}"
protocol: TCP
targetPort: "{{ .Values.ports.airflowUI }}"
loadBalancerIP: ~
loadBalancerSourceRanges: []
nodeSelector: {}
priorityClassName: ~
affinity: {}
tolerations: []
topologySpreadConstraints: []
podAnnotations: {}
labels: {}
waitForMigrations:
enabled: true
env: []
env: []
# Airflow Triggerer Config
triggerer:
enabled: false
# Airflow Dag Processor Config
dagProcessor:
enabled: false
# Flower settings
flower:
enabled: false
# StatsD settings
statsd:
enabled: true
revisionHistoryLimit: ~
serviceAccount:
create: true
name: ~
annotations: {}
uid: 65534
securityContext: {}
# Additional network policies as needed
extraNetworkPolicies: []
resources:
limits:
cpu: 3000m
memory: 5Gi
requests:
cpu: 2000m
memory: 3Gi
service:
extraAnnotations: {}
nodeSelector: {}
affinity: {}
tolerations: []
topologySpreadConstraints: []
priorityClassName: ~
extraMappings: []
overrideMappings: []
podAnnotations: {}
# PgBouncer settings
pgbouncer:
enabled: false
# Configuration for the redis provisioned by the chart
redis:
enabled: false
terminationGracePeriodSeconds: 600
serviceAccount:
create: false
name: airflow-redis
annotations: {}
persistence:
enabled: false
size: 1Gi
storageClassName:
annotations: {}
resources:
limits:
cpu: 300m
memory: 16Mi
requests:
cpu: 100m
memory: 8Mi
passwordSecretName: ~
password: ~
safeToEvict: true
nodeSelector: {}
affinity: {}
tolerations: []
topologySpreadConstraints: []
uid: 0
securityContext: {}
podAnnotations: {}
# Auth secret for a private registry
registry:
secretName: ~
connection: {}
# Elasticsearch logging configuration
elasticsearch:
enabled: false
# All ports used by chart
ports:
flowerUI: 5555
airflowUI: 8080
workerLogs: 8793
redisDB: 6379
statsdIngest: 9125
statsdScrape: 9102
pgbouncer: 6543
pgbouncerScrape: 9127
# Define any ResourceQuotas for namespace
quotas: {}
# Define default/max/min values for pods and containers in namespace
limits: []
# This runs as a CronJob to cleanup old pods.
cleanup:
enabled: false
# Configuration for postgresql subchart
# Not recommended for production
postgresql:
enabled: false
# Config settings to go into the mounted airflow.cfg
#
# Please note that these values are passed through the `tpl` function, so are
# all subject to being rendered as go templates. If you need to include a
# literal `{{` in a value, it must be expressed like this:
#
# a: '{{ "{{ not a template }}" }}'
#
# Do not set config containing secrets via plain text values, use Env Var or k8s secret object
# yamllint disable rule:line-length
config:
core:
dags_folder: '{{ include "airflow_dags" . }}'
# This is ignored when used with the official Docker image
load_examples: 'False'
executor: '{{ .Values.executor }}'
# For Airflow 1.10, backward compatibility; moved to [logging] in 2.0
colored_console_log: 'False'
remote_logging: '{{- ternary "True" "False" .Values.elasticsearch.enabled }}'
logging:
remote_logging: '{{- ternary "True" "False" .Values.elasticsearch.enabled }}'
colored_console_log: 'False'
metrics:
statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
statsd_port: 9125
statsd_prefix: airflow
statsd_host: '{{ printf "%s-statsd" .Release.Name }}'
webserver:
enable_proxy_fix: 'True'
# For Airflow 1.10
rbac: 'True'
worker_class: 'sync'
update_fab_perms: 'True'
celery:
flower_url_prefix: '{{ .Values.ingress.flower.path }}'
worker_concurrency: 16
scheduler:
standalone_dag_processor: '{{ ternary "True" "False" .Values.dagProcessor.enabled }}'
# statsd params included for Airflow 1.10 backward compatibility; moved to [metrics] in 2.0
statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
statsd_port: 9125
statsd_prefix: airflow
statsd_host: '{{ printf "%s-statsd" .Release.Name }}'
# `run_duration` included for Airflow 1.10 backward compatibility; removed in 2.0.
run_duration: 41460
elasticsearch:
json_format: 'True'
log_id_template: "{dag_id}_{task_id}_{execution_date}_{try_number}"
elasticsearch_configs:
max_retries: 3
timeout: 30
retry_timeout: 'True'
kerberos:
keytab: '{{ .Values.kerberos.keytabPath }}'
reinit_frequency: '{{ .Values.kerberos.reinitFrequency }}'
principal: '{{ .Values.kerberos.principal }}'
ccache: '{{ .Values.kerberos.ccacheMountPath }}/{{ .Values.kerberos.ccacheFileName }}'
celery_kubernetes_executor:
kubernetes_queue: 'kubernetes'
kubernetes:
namespace: '{{ .Release.Namespace }}'
airflow_configmap: '{{ include "airflow_config" . }}'
airflow_local_settings_configmap: '{{ include "airflow_config" . }}'
pod_template_file: '{{ include "airflow_pod_template_file" . }}/pod_template_file.yaml'
worker_container_repository: '{{ .Values.images.airflow.repository | default .Values.defaultAirflowRepository }}'
worker_container_tag: '{{ .Values.images.airflow.tag | default .Values.defaultAirflowTag }}'
multi_namespace_mode: '{{ ternary "True" "False" .Values.multiNamespaceMode }}'
database:
sql_alchemy_connect_args: airflow_local_settings.keepalive_kwargs
multiNamespaceMode: false
podTemplate: ~
# Git sync
dags:
persistence:
enabled: true
size: 5Gi
storageClassName:
accessMode: ReadWriteMany
existingClaim: "xxxxxxxx"
subPath: ~
gitSync:
enabled: false
logs:
persistence:
enabled: false
size: 100Gi
storageClassName:
existingClaim:
# CSI Fuse
storageClass:
name: csi-gcs-fuse-data-eng
`
### Operating System
linux
### Versions of Apache Airflow Providers
2.10.5
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Airflow Version:
Originally 2.9.2, upgraded to 2.10.5 (issue persists in both versions)
Executor:
KubernetesExecutor
Orchestration Platform:
Google Kubernetes Engine (GKE) – Standard mode (not Autopilot)
Cluster Size & Scaling:
~20 nodes (n2-highmem-8)
Autoscaling enabled, but cluster consistently runs at <50% CPU/memory usage
Nodes use standard nodepools, not preemptible or spot instances
Airflow Deployment Mode:
Deployed via Helm with custom image (Python 3.10 base)
Webserver, Scheduler, and Triggerer run as separate pods
Using Google Cloud Composer-like architecture, but self-managed
Task Workloads:
~1200 DAGs, mostly scheduled daily/hourly
~100–150 concurrent tasks at peak times
Heavy usage of:
DatabricksSubmitRunOperator
SnowflakeOperator
ExternalTaskSensor
ShortCircuitOperator
### Anything else?
We have thoroughly verified that this is not caused by:
- Resource exhaustion (CPU, memory, or disk)
- Pod evictions or node preemptions
- Task timeouts or DAG-level retries
- Manual task terminations
- Implemented dag timeout and execution timeout also created separate pool for databricks and snowflake task
We suspect the issue may stem from one of the following:
-Internal Airflow lifecycle logic (e.g., zombie detection, pod orphan cleanup)
- K8sExecutor edge cases where the scheduler or triggerer may initiate SIGTERM without resource-based justification
- GCSFuse or sidecar instability possibly impacting task pod lifecycle (though main containers are healthy)
- We are open to instrumenting Airflow internals with logging or tracing if the core team can suggest areas to probe (e.g: executor heartbeat, cleanup routines, orphan detection)
We would also be happy to help test a patch or proposed fix in our production-like test cluster if needed.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected SIGTERM on Tasks (Airflow 2.10.5 on GKE with KubernetesExecutor and No Resource Constraints) [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #53894:
URL: https://github.com/apache/airflow/issues/53894#issuecomment-3133369635
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected SIGTERM on Tasks (Airflow 2.10.5 on GKE with KubernetesExecutor and No Resource Constraints) [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #53894: Unexpected SIGTERM on Tasks (Airflow 2.10.5 on GKE with KubernetesExecutor and No Resource Constraints)
URL: https://github.com/apache/airflow/issues/53894
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected SIGTERM on Tasks (Airflow 2.10.5 on GKE with KubernetesExecutor and No Resource Constraints) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #53894:
URL: https://github.com/apache/airflow/issues/53894#issuecomment-3139162455
I think your best bet is to attempt to upgrade to Airflow 3. This is issue might take forever to diagnose because there is absolutely no clue on why it happens. The whole mechanism of heartbeating, supervising running tasks, etc. that could be causeing it had been completely rewritten there and I think the fastest way to get this problem solved is to simply switch to Airflow 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org