[GitHub] [airflow] patrickbrady-xaxis commented on issue #16982: Tasks fail and do not log due to backend DB (dead?)lock

[GitBox <gi...@apache.org>]

patrickbrady-xaxis commented on issue #16982:
URL: https://github.com/apache/airflow/issues/16982#issuecomment-1035050661


   @potiuk to add some additional color, my team and I just patched what seemed to be a very similar issue to the OP (deadlocks during xcom table updates) on our own DB by adding a separate index to our xcom table on `dag_id`, `task_id`, and `execution_date`. A similar index had been present in 2.1.2 but was removed when we migrated to 2.1.4.
   
   Our specific implementation runs thousands of the same dag_ids simultaneously with different configurations, so with only the primary key on `dag_id`, `task_id`, `key`, and `execution_date`, every xcom update query was only able to narrow as far as `dag_id + task_id`, leaving thousands of rows to scan for a matching `execution_date`. All of our tasks update xcom with status codes, and many of the tasks have similar run times across different dag runs, leading to large numbers of concurrent requests with `execution_date` as the only distinguishing factor.
   
   Hope that might be helpful to anyone with a similar setup.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org