You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jobs@airflow.apache.org by GitBox <gi...@apache.org> on 2026/01/25 07:13:37 UTC
[GH] (airflow/fix/kpo-aws-cli-cache-race-condition): Workflow run "Tests" failed!
The GitHub Actions job "Tests" on airflow.git/fix/kpo-aws-cli-cache-race-condition has failed.
Run started by GitHub user Arunodoy18 (triggered by Arunodoy18).
Head commit for run:
17bf3c26000121414ef712c89b1ea108ef17112c / Arunodoy18 <ar...@gmail.com>
Fix AWS CLI cache race condition in KubernetesPodOperator
When multiple KubernetesPodOperator tasks run in parallel on the same
Celery worker with EKS authentication, they concurrently invoke the AWS
CLI for token generation. This causes a race condition where all tasks
attempt to create ~/.aws/cli/cache simultaneously, resulting in
FileExistsError and task failure before pod creation.
This commit adds AWS CLI cache directory isolation per task execution:
- Created _isolated_aws_cli_cache() context manager in kubernetes hook
- Sets task-specific AWS_SHARED_CREDENTIALS_FILE env var with UUID
- Wraps all config.load_kube_config() calls with the isolation context
- Ensures cleanup of temporary directories after authentication
- Added concurrent authentication test to verify no race conditions
The fix is transparent to users - no configuration changes required.
Non-EKS and in-cluster setups are unaffected. Each parallel KPO task
now uses its own isolated AWS cache directory, preventing filesystem
collisions during concurrent EKS authentication.
Fixes race condition causing FileExistsError in parallel KPO execution.
Report URL: https://github.com/apache/airflow/actions/runs/21328107844
With regards,
GitHub Actions via GitBox
---------------------------------------------------------------------
To unsubscribe, e-mail: jobs-unsubscribe@airflow.apache.org
For additional commands, e-mail: jobs-help@airflow.apache.org