You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Nunemunthala Sudheer (JIRA)" <ji...@apache.org> on 2017/12/18 11:39:00 UTC
[jira] [Commented] (AIRFLOW-1511) jobs failed ever in backfill due
to concurrency limit
[ https://issues.apache.org/jira/browse/AIRFLOW-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16294856#comment-16294856 ]
Nunemunthala Sudheer commented on AIRFLOW-1511:
-----------------------------------------------
Hi [~xianping],
We are also facing similar issue, did you got any solution?
> jobs failed ever in backfill due to concurrency limit
> -----------------------------------------------------
>
> Key: AIRFLOW-1511
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1511
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill
> Affects Versions: Airflow 1.8
> Reporter: Xianping lin
>
> backfill a DAG which supposed to run daily. some jobs can't start forever. this is in cluster mode by celery executor.
> current time is : Tue Aug 15 04:56:56 PDT 2017.
> Those daily job still can't start. see below messages:
> *** Log file isn't local.
> *** Fetching here: http://PCDM3:8793/log/user_subs/user_subs_calc/2017-07-08T00:00:00
> [2017-08-14 22:11:36,700] {cli.py:391} INFO - Loading pickle id 205
> [2017-08-14 22:11:36,818] {base_task_runner.py:112} INFO - Running: ['bash', '-c', u'airflow run user_subs user_subs_calc 2017-07-08T00:00:00 --pickle 205 --job_id 21317 --raw']
> [2017-08-14 22:11:37,723] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:37,723] {__init__.py:57} INFO - Using executor CeleryExecutor
> [2017-08-14 22:11:37,878] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:37,877] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2017-08-14 22:11:37,911] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:37,911] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2017-08-14 22:11:38,176] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:38,176] {cli.py:391} INFO - Loading pickle id 205
> [2017-08-14 22:11:38,249] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:38,249] {models.py:1126} INFO - Dependencies all met for <TaskInstance: user_subs.user_subs_calc 2017-07-08 00:00:00 [queued]>
> [2017-08-14 22:11:38,253] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:38,253] {models.py:1120} INFO - Dependencies not met for <TaskInstance: user_subs.user_subs_calc 2017-07-08 00:00:00 [queued]>, dependency 'Task Instance Slots Available' FAILED: The maximum number of running tasks (user_subs) for this task's DAG '16' has been reached.
> [2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:38,262] {models.py:1300} WARNING -
> [2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
> [2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask: FIXME: Rescheduling due to concurrency limits reached at task runtime. Attempt 1 of 4. State set to NONE.
> [2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
> [2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask:
> [2017-08-14 22:11:38,263] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:38,262] {models.py:1304} INFO - Queuing into pool None
> [2017-08-14 22:11:41,822] {jobs.py:2083} INFO - Task exited with return code 0
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)