[GitHub] [airflow] MatrixManAtYrService opened a new issue #17901: KubernetesExecutor: kubectl task gets stuck running if command takes more than 0.4 sec

[GitBox <gi...@apache.org>]

MatrixManAtYrService opened a new issue #17901:
URL: https://github.com/apache/airflow/issues/17901


   ### Apache Airflow version
   
   2.1.3 (latest released)
   
   ### Operating System
   
   20.04.1-Ubuntu
   
   ### Versions of Apache Airflow Providers
   
   Providers info
   apache-airflow-providers-amazon          | 2.1.0
   apache-airflow-providers-celery          | 2.0.0
   apache-airflow-providers-cncf-kubernetes | 2.0.2
   apache-airflow-providers-docker          | 2.1.0
   apache-airflow-providers-elasticsearch   | 2.0.2
   apache-airflow-providers-ftp             | 2.0.0
   apache-airflow-providers-google          | 5.0.0
   apache-airflow-providers-grpc            | 2.0.0
   apache-airflow-providers-hashicorp       | 2.0.0
   apache-airflow-providers-http            | 2.0.0
   apache-airflow-providers-imap            | 2.0.0
   apache-airflow-providers-microsoft-azure | 3.1.0
   apache-airflow-providers-mysql           | 2.1.0
   apache-airflow-providers-postgres        | 2.0.0
   apache-airflow-providers-redis           | 2.0.0
   apache-airflow-providers-sendgrid        | 2.0.0
   apache-airflow-providers-sftp            | 2.1.0
   apache-airflow-providers-slack           | 4.0.0
   apache-airflow-providers-sqlite          | 2.0.0
   apache-airflow-providers-ssh             | 2.1.0
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   Dockerfile:
   ```
   FROM apache/airflow:2.1.3
   USER root
   RUN apt update && apt install -y curl
   
   # kubectl
   RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   RUN install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
   
   # kubeconfig
   RUN mkdir -p ~/.kube
   COPY ./config /home/airflow/.kube/config
   ENV KUBECONFIG=/home/airflow/.kube/config
   RUN chown airflow /home/airflow/.kube/config
   
   COPY increment.py $AIRFLOW_HOME/dags/increment.py
   
   USER airflow
   
   RUN pip install sh
   ```
   
   values.yaml:
   ```
   executor: KubernetesExecutor
   defaultAirflowRepository: 192.168.90.13:30500/airflow
   defaultAirflowTag: freezebug
   images:
     airflow:
       pullPolicy: Always
     flower:
       pullPolicy: Always
     pod_template:
       pullPolicy: Always
   logs:
     persistence:
       enabled: true
       size: 1Gi
   ```
   
   ### What happened
   
   I had defined a DAG which uses `kubectl run` to run `bash -c 'sleep X'` for successively higher values of X:
   - 0 sec
   - 0.1 sec
   - 0.2 sec
   - ...
   - 2 sec
   
   Around 0.3 seconds, the tasks stopped completing:
   <img width="381" alt="stuck_eventually" src="https://user-images.githubusercontent.com/5834582/131287433-a8abaf1d-cfaa-4bcd-93ef-e754b20c41de.png">
   
   
   ### What you expected to happen
   
   I expected the tasks to all complete successfully
   
   ### How to reproduce
   
   I made[ a repo](https://github.com/MatrixManAtYrService/bug_airflowfreeze) with the files that I used.  See [commands.txt](https://github.com/MatrixManAtYrService/bug_airflowfreeze/blob/main/commands.txt) for the flow.
   
   Generally:
   - docker build an image with the problematic DAG, include kubectl and a kubeconfig file for a running cluster
   - push the image to a repository (I use https://github.com/twuni/docker-registry.helm to deploy one locally)
   - deploy Airflow in that image with the airflow helm chart
   - run the "increment" DAG
   - notice that it never completes
   
   ### Anything else
   
   If you use the Sequential Executor, it completes just fine
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org