You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/04 12:38:20 UTC

[GitHub] [airflow] madcolonel10 opened a new issue #12089: Task of dag stuck in queued state forever when using kubernetes executor and triggered with a future execution date

madcolonel10 opened a new issue #12089:
URL: https://github.com/apache/airflow/issues/12089


   **Airflow version: 1.10.12**
   ```python
   import random
   import string
   
   import requests
   from airflow import DAG
   from airflow.contrib.sensors.python_sensor import PythonSensor
   from airflow.operators.dummy_operator import DummyOperator
   from datetime import datetime
   
   dag = DAG(
       dag_id='newDebugDag',
       description='A simple tutorial DAG',
       schedule_interval=None,
       start_date=datetime(2020, 10, 30, 0, 0)
   )
   
   start = DummyOperator(task_id='run_this_first', dag=dag)
   
   
   def wait_for_approval(dag_run, **context):
       software_id = dag_run.conf.get("softwareId", "Some Software Id")
       print("polling status for softwareId:{}".format(software_id))
       response = requests.get('https://5f8582eec29abd0016190be2.mockapi.io/api/v1/status')
       print(response.json())
       context["task_instance"].xcom_push(key="approverId", value="visardan-" + random.choice(string.ascii_letters))
       return response.json()[0]['status']
   
   
   wait_for_approval = PythonSensor(
       task_id="wait_for_approval_debug",
       python_callable=wait_for_approval,
       dag=dag,
       poke_interval=30,
       provide_context=True,
       executor_config={"KubernetesExecutor": {"image": "apache/airflow:1.10.12-python3.6"}}
   )
   
   start >> wait_for_approval
   ```
   
   ```json
   {
       "run_id":"run-eventId109bnfghak2",
       "conf":{
           "softwareId":"something"
       },
       "execution_date": "2020-11-04 10:38:43" //any future execution date
   }
   ```
   
   The first task executes fine but the dag is stuck at second task. I can see the worker pod spun up and getting completed but the task status never gets updated and ui still says queued and dag is always in running state due to this.
   
   ![Screen Shot 2020-11-04 at 5 02 22 PM](https://user-images.githubusercontent.com/25412835/98112292-f7398800-1ec7-11eb-978d-ada384b0ac81.png)
   
   ![Screen Shot 2020-11-04 at 5 04 47 PM](https://user-images.githubusercontent.com/25412835/98112298-fbfe3c00-1ec7-11eb-8cea-a1699d5b03cc.png)
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] JieChenAtPonyai edited a comment on issue #12089: Task of dag stuck in queued state forever when using kubernetes executor, and triggered with a future execution date

Posted by GitBox <gi...@apache.org>.
JieChenAtPonyai edited a comment on issue #12089:
URL: https://github.com/apache/airflow/issues/12089#issuecomment-796772246


   I met a similar issue. After investigation, I find the root cause is the the worker pod does not update the status of task to RUNNING due to this dependence [RunnableExecDateDep](https://github.com/apache/airflow/blob/master/airflow/ti_deps/deps/runnable_exec_date_dep.py) is not met. So this task is stuck in queued state.
   I think enabling [allow_trigger_in_future]（https://github.com/apache/airflow/blob/f2558cdd55ee4900c7f7f0c300354c742cd38d8c/airflow/config_templates/default_airflow.cfg#L928) can resolve it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] JieChenAtPonyai edited a comment on issue #12089: Task of dag stuck in queued state forever when using kubernetes executor, and triggered with a future execution date

Posted by GitBox <gi...@apache.org>.
JieChenAtPonyai edited a comment on issue #12089:
URL: https://github.com/apache/airflow/issues/12089#issuecomment-796772246


   I met a similar issue. After investigation, I find the root cause is the the worker pod does not update the status of task to RUNNING because this dependence [RunnableExecDateDep](https://github.com/apache/airflow/blob/master/airflow/ti_deps/deps/runnable_exec_date_dep.py) is not met. So this task is stuck in queued state.
   I think enabling [allow_trigger_in_future]（https://github.com/apache/airflow/blob/f2558cdd55ee4900c7f7f0c300354c742cd38d8c/airflow/config_templates/default_airflow.cfg#L928) can resolve it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] JieChenAtPonyai commented on issue #12089: Task of dag stuck in queued state forever when using kubernetes executor, and triggered with a future execution date

Posted by GitBox <gi...@apache.org>.
JieChenAtPonyai commented on issue #12089:
URL: https://github.com/apache/airflow/issues/12089#issuecomment-796772246


   I met a similar issue. After investigation, I find the root cause is the the worker pod does not update the status of task to RUNNING due to this dependence [RunnableExecDateDep](https://github.com/apache/airflow/blob/master/airflow/ti_deps/deps/runnable_exec_date_dep.py) is not met. So this task is stuck in queued state.
   I think enabling [allow_trigger_in_future])https://github.com/apache/airflow/blob/f2558cdd55ee4900c7f7f0c300354c742cd38d8c/airflow/config_templates/default_airflow.cfg#L928) can resolve it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #12089: Task of dag stuck in queued state forever when using kubernetes executor and triggered with a future execution date

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12089:
URL: https://github.com/apache/airflow/issues/12089#issuecomment-721707739


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] grepthat commented on issue #12089: Task of dag stuck in queued state forever when using kubernetes executor, and triggered with a future execution date

Posted by GitBox <gi...@apache.org>.
grepthat commented on issue #12089:
URL: https://github.com/apache/airflow/issues/12089#issuecomment-777262942


   duplicate of https://github.com/apache/airflow/issues/13542 ? Are you still seeing this in Airflow 2.0 ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org