Task moving from RUNNING to SUCCEEDED to REMOVED

[Nasron Cheong <na...@gmail.com>]

Hi,

I'm trying to debug an issue where a task in my dag is being set to REMOVED
state, and the tasks after are running waiting for preceding tasks to
succeed/fail.

If I delete the removed tasks from the meta database, the tasks will start
running, move to succeeded state, but then promptly move to REMOVED state
again.

A change that may have caused this was to change the dag from:

latest_only >> cleanup_hdfs >> run_mr
run_mr >> refresh_tables >> refresh_done_file
run_mr >> refresh_global_table >> refresh_done_file
refresh_done_file >> reap_old

to:

latest_only >> cleanup_hdfs >> run_mr
run_mr >> refresh_tables >> *compute_stats_orgs* >> refresh_done_file
run_mr >> refresh_global_table >> *compute_stats_global* >>
refresh_done_file
refresh_done_file >> reap_old

Where the highlighted tasks were added.

Any idea what would cause this?

I've tried clearing all task instances for the dag, restarting all
services, no luck.

I'm on 1.9.0

Thanks!

- Nasron