[GitHub] [airflow] dstandish opened a new issue, #24974: Handle potential write conflict in queue table

[GitBox <gi...@apache.org>]

dstandish opened a new issue, #24974:
URL: https://github.com/apache/airflow/issues/24974

   Currently the DatasetDagRunQueue table is one record per target_dag_id+dataset_id.
   This is because we don't care if a dataset is updated multiple times in between triggered dag runs (e.g. while waiting on _another_ dataset to be updated -- once is enough information.
   BUT, the problem is, sqlalchemy's merge statement doesn't actually do a db-level merge, which means that it is not concurrency-safe (i.e. it's susceptible to race condition if another process wants to add the same queue record). 
   
   So, we either need to split TI "success" and queue-table-write into separate commits, or we need to loosen the grain of the queue table so that we can always add more queue records even when more won't make a difference.  Or, thirdly, I guess we could try and figure out a way to do a safe insert-or-update.
   
   Ash initially specified that success + queue should be in same transaction, so unless there's a good reason we should prioritize the other two options.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish closed issue #24974: [Won't do] Maybe handle potential [unlikely] write conflict in queue table?

[GitBox <gi...@apache.org>]

dstandish closed issue #24974: [Won't do] Maybe handle potential [unlikely] write conflict in queue table?
URL: https://github.com/apache/airflow/issues/24974


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org