You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Michael Crawford (JIRA)" <ji...@apache.org> on 2018/09/23 11:10:00 UTC
[jira] [Commented] (AIRFLOW-1482) Error when try to backfill the
example_trigger_controller_dag
[ https://issues.apache.org/jira/browse/AIRFLOW-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16625069#comment-16625069 ]
Michael Crawford commented on AIRFLOW-1482:
-------------------------------------------
Ran into this issue as well. It appears to be a pickle issue (atleast in our case) We were able to work around it by turning donot_pickle to True.
We only had this issue when using celery.
> Error when try to backfill the example_trigger_controller_dag
> -------------------------------------------------------------
>
> Key: AIRFLOW-1482
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1482
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill
> Affects Versions: 1.8.1, 1.8.2
> Environment: Ubuntu: 16.04
> Python: 2.7
> CeleryExecutor
> Broker: Redis
> Reporter: Timothee N
> Priority: Blocker
> Attachments: airflow_1.png, airflow_2.png, airflow_3.png
>
>
> Hello,
> Running a backfill command for the {noformat}example_trigger_controller_dag{noformat} example dag, result in the failed task {noformat}test_trigger_dagrun{noformat}
> It seems to me that the problem comes from the TriggerDagRunOperator in the example_trigger_controller_dag ?
> Backfill command: {noformat}airflow backfill -s 2017-07-10 -e 2017-07-13 --pool backfill example_trigger_controller_dag{noformat}
> Tested in 1.8.1 and 1.8.2rc1
> Here is the output log from the backfill command :
> {noformat}
> [2017-08-02 13:53:00,844] {__init__.py:57} INFO - Using executor CeleryExecutor
> [2017-08-02 13:53:00,888] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2017-08-02 13:53:00,902] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> /var/lib/airflow/local/lib/python2.7/site-packages/airflow/www/app.py:23: FlaskWTFDeprecationWarning: "flask_wtf.CsrfProtect" has been renamed to "CSRFProtect" and will be removed in 1.0.
> csrf = CsrfProtect()
> [2017-08-02 13:53:01,033] {models.py:168} INFO - Filling up the DagBag from /var/lib/airflow/dags
> [2017-08-02 13:53:01,332] {models.py:1128} INFO - Dependencies all met for <TaskInstance: example_trigger_controller_dag.test_trigger_dagrun 2017-07-10 00:00:00 [scheduled]>
> [2017-08-02 13:53:01,337] {base_executor.py:50} INFO - Adding to queue: airflow run example_trigger_controller_dag test_trigger_dagrun 2017-07-10T00:00:00 --pickle 1 --local --pool backfill
> [2017-08-02 13:53:06,267] {celery_executor.py:81} INFO - [celery] queuing (u'example_trigger_controller_dag', u'test_trigger_dagrun', datetime.datetime(2017, 7, 10, 0, 0)) through celery, queue=default
> [2017-08-02 13:53:06,330] {models.py:4164} INFO - Updating state for <DagRun example_trigger_controller_dag @ 2017-07-10 00:00:00: backfill_2017-07-10T00:00:00, externally triggered: False> considering 1 task(s)
> [2017-08-02 13:53:06,334] {jobs.py:2020} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 0 | succeeded: 0 | kicked_off: 1 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
> [2017-08-02 13:53:11,273] {jobs.py:1743} ERROR - Executor reports task instance <TaskInstance: example_trigger_controller_dag.test_trigger_dagrun 2017-07-10 00:00:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally?
> [2017-08-02 13:53:11,273] {models.py:1433} ERROR - Executor reports task instance <TaskInstance: example_trigger_controller_dag.test_trigger_dagrun 2017-07-10 00:00:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally?
> None
> [2017-08-02 13:53:11,273] {models.py:1457} INFO - Marking task as FAILED.
> [2017-08-02 13:53:11,279] {models.py:1478} ERROR - Executor reports task instance <TaskInstance: example_trigger_controller_dag.test_trigger_dagrun 2017-07-10 00:00:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally?
> [2017-08-02 13:53:11,281] {jobs.py:1694} ERROR - Task instance <TaskInstance: example_trigger_controller_dag.test_trigger_dagrun 2017-07-10 00:00:00 [failed]> failed
> [2017-08-02 13:53:11,283] {models.py:4164} INFO - Updating state for <DagRun example_trigger_controller_dag @ 2017-07-10 00:00:00: backfill_2017-07-10T00:00:00, externally triggered: False> considering 1 task(s)
> [2017-08-02 13:53:11,285] {models.py:4204} INFO - Marking run <DagRun example_trigger_controller_dag @ 2017-07-10 00:00:00: backfill_2017-07-10T00:00:00, externally triggered: False> failed
> [2017-08-02 13:53:11,298] {jobs.py:2020} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 0 | kicked_off: 0 | failed: 1 | skipped: 0 | deadlocked: 0 | not ready: 0
> Traceback (most recent call last):
> File "/var/lib/airflow/bin/airflow", line 28, in <module>
> args.func(args)
> File "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 167, in backfill
> pool=args.pool)
> File "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/models.py", line 3373, in run
> job.run()
> File "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/jobs.py", line 201, in run
> self._execute()
> File "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/jobs.py", line 2063, in _execute
> raise AirflowException(err)
> airflow.exceptions.AirflowException: ---------------------------------------------------
> Some task instances failed:
> set([(u'example_trigger_controller_dag', u'test_trigger_dagrun', datetime.datetime(2017, 7, 10, 0, 0))])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)