[GitHub] [airflow] jedcunningham opened a new issue #15218: Task stuck in queued state with pending pod

[GitBox <gi...@apache.org>]

jedcunningham opened a new issue #15218:
URL: https://github.com/apache/airflow/issues/15218


   **Apache Airflow version**: 2.0.1
   **Kubernetes version**: v1.19.7
   **Executor**: KubernetesExecutor
   
   **What happened**:
   
   If you have a worker that gets stuck in pending forever, say with a missing volume mount, the task will stay in the queued state forever. Nothing is applying a timeout on it actually being able to start.
   
   **What you expected to happen**:
   
   Eventually the scheduler will notice that the worker hasn't progressed past pending after a given amount of time and will mark it as a failure.
   
   **How to reproduce it**:
   ```python
   from airflow import DAG
   from airflow.operators.bash import BashOperator
   from airflow.utils.dates import days_ago
   from kubernetes.client import models as k8s
   
   
   default_args = {
       "owner": "airflow",
   }
   
   with DAG(
       dag_id="pending",
       default_args=default_args,
       schedule_interval=None,
       start_date=days_ago(2),
   ) as dag:
       BashOperator(
           task_id="forever_pending",
           bash_command="date; sleep 30; date",
           executor_config={
               "pod_override": k8s.V1Pod(
                   spec=k8s.V1PodSpec(
                       containers=[
                           k8s.V1Container(
                               name="base",
                               volume_mounts=[
                                   k8s.V1VolumeMount(mount_path="/foo/", name="vol")
                               ],
                           )
                       ],
                       volumes=[
                           k8s.V1Volume(
                               name="vol",
                               persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(
                                   claim_name="missing"
                               ),
                           )
                       ],
                   )
               ),
           },
       )
   ```
   
   
   **Anything else we need to know**:
   Related to:
   
   * #15149 (This is reporting that these pending pods don't get deleted via "Mark Failed")
   * #14556 (This handles when these pending pods get deleted and is already fixed)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil closed issue #15218: Task stuck in queued state with pending pod

[GitBox <gi...@apache.org>]

kaxil closed issue #15218:
URL: https://github.com/apache/airflow/issues/15218


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org