[jira] [Updated] (AIRFLOW-4496) Airflow `backfill` fails on pickle thread error when --task_regex used

["Trevor Burke (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Trevor Burke updated AIRFLOW-4496:
----------------------------------
    Environment: 
Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-1075-aws x86_64)
Postgres LocalExecutor

  was:Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-1075-aws x86_64)


> Airflow `backfill` fails on pickle thread error when --task_regex used
> ----------------------------------------------------------------------
>
>                 Key: AIRFLOW-4496
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4496
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: DAG
>    Affects Versions: 1.10.2
>         Environment: Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-1075-aws x86_64)
> Postgres LocalExecutor
>            Reporter: Trevor Burke
>            Priority: Major
>
> Airflow backfill works properly when used without task_regex, but when I employ that flog I get the following stack trace:
> {code:java}
> TypeError: can't pickle _thread.RLock objects
> {code}
> The command I'm using is:
> {code:java}
> airflow backfill <dag_id>  -s 2019-04-15 -e 2019-05-08 -x -t normalize -i --reset_dagruns
> {code}
> {code:python}
> interval_args = {
>     'owner': 'airflow',
>     'depends_on_past': True,
>     'start_date': datetime(2019, 4, 15),
>     'retries': 2,
>     'retry_delay': timedelta(minutes=5),
>     'on_failure_callback': send_email
> }
> interval_dag = DAG('dag_id_redacted',
>                    default_args=interval_args,
>                    schedule_interval='*/15 * * * *',
>                    catchup=True,
>                    user_defined_macros=dict(DBT=DBT),)
> {code}
> The task flow is basically get data from external API, dump to S3, flatten for database loading, and load to database. The tasks have been performing perfectly fine and previous backfills have been successful, but task_regex has been giving me issues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)