You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/19 05:54:13 UTC

[GitHub] [airflow] atishkathpal commented on issue #20047: ERROR: could not queue task TaskInstanceKey

atishkathpal commented on issue #20047:
URL: https://github.com/apache/airflow/issues/20047#issuecomment-1016109287


   Even I am hitting this error however I have configured task retries which ensures that another try is attempted, which usually succeeds. I am not using ExternalTaskSensor though. 
   
   In my case I have ~80 DAGs running every minute, and ~13 tasks tend to hit this issue but end up successfully completing on 2nd attempt. I have sufficient workers and worker concurrency. I have sufficient schedulers as well, so it's not clear what is causing such an intermittent failure to queue the task, such failures result in delayed completion of DAGs. 
   
   I have over 20K DAGs in the cluster and using KubernetesCeleryExecutor. 
   
   > [[34m2022-01-19 04:41:30,514[0m] {[34mbase_executor.py:[0m85} ERROR[0m - could not queue task TaskInstanceKey(dag_id='931_dag_3761', task_id='task_2', run_id='scheduled__2022-01-19T03:41:00+00:00', try_number=1)[0m


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org