[jira] [Created] (AIRFLOW-4496) Airflow `backfill` fails on pickle thread error when --task_regex used

["Trevor Burke (JIRA)" <ji...@apache.org>]

Trevor Burke created AIRFLOW-4496:
-------------------------------------

             Summary: Airflow `backfill` fails on pickle thread error when --task_regex used
                 Key: AIRFLOW-4496
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4496
             Project: Apache Airflow
          Issue Type: Bug
          Components: DAG
    Affects Versions: 1.10.2
         Environment: Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-1075-aws x86_64)
            Reporter: Trevor Burke


Airflow backfill works properly when used without task_regex, but when I employ that flog I get the following stack trace:
{code:java}
TypeError: can't pickle _thread.RLock objects
{code}

The command I'm using is:

{code:java}
airflow backfill <dag_id>  -s 2019-04-15 -e 2019-05-08 -x -t normalize -i --reset_dagruns
{code}

{code:python}
interval_args = {
    'owner': 'airflow',
    'depends_on_past': True,
    'start_date': datetime(2019, 4, 15),
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
    'on_failure_callback': send_email
}

interval_dag = DAG('dag_id_redacted',
                   default_args=interval_args,
                   schedule_interval='*/15 * * * *',
                   catchup=True,
                   user_defined_macros=dict(DBT=DBT),)

{code}


The task flow is basically get data from external API, dump to S3, flatten for database loading, and load to database. The tasks have been performing perfectly fine and previous backfills have been successful, but task_regex has been giving me issues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)