[jira] [Created] (AIRFLOW-657) MsSqlOperator Autocommit Option

["Robin Miller (JIRA)" <ji...@apache.org>]

Robin Miller created AIRFLOW-657:
------------------------------------

             Summary: MsSqlOperator Autocommit Option
                 Key: AIRFLOW-657
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-657
             Project: Apache Airflow
          Issue Type: Improvement
          Components: operators
            Reporter: Robin Miller
            Assignee: Robin Miller
            Priority: Trivial


The MsSqlOperator is not exposing the option to enable autocommit, even though the MsSqlHook has supports_autocommit to true.

This means that tasks running on MsSql are always wrapped in an additional transaction, which means that parallel tasks using shared resources can cause SQL Deadlocks.

We see no reason not to expose this option to the user via the operator and have produced and tested the change, keeping the existing behaviour as the default.

In our use case we have a number of identical parallel tasks to work through a list of identical build tasks with an appropriate level of parallelism. This is prevented from working as intended since the first of these tasks to start, updates this task list and then holds a transaction lock on this table, meaning the other threads cannot then access the table. (It would not be convenient to have a task for each task in the list defined in the dag as there are upwards of 7000 tasks and it is convenient for us to be able to manually add or remove tasks from this list during execution)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)