incubator-airflow git commit: [AIRFLOW-1937] Speed up scheduling by committing in batch

[fo...@apache.org]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master c0dffb57c -> cc62fd064


[AIRFLOW-1937] Speed up scheduling by committing in batch

Newly scheduled task instances (state = None, up
for retry)
were committed per task instance instead of all at
once.
This isn't required as tasks cannot be picked up
by another
process in the mean time. Committing in batch
significantly
speeds up task scheduling for dags that have a lot
of tasks.

Closes #2888 from bolkedebruin/AIRFLOW-1937


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/cc62fd06
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/cc62fd06
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/cc62fd06

Branch: refs/heads/master
Commit: cc62fd064a02f7d14d2a0e03466b8bd424fd6f30
Parents: c0dffb5
Author: Bolke de Bruin <bo...@xs4all.nl>
Authored: Sat Dec 23 09:41:36 2017 +0100
Committer: Fokko Driesprong <fo...@godatadriven.com>
Committed: Sat Dec 23 09:41:36 2017 +0100

----------------------------------------------------------------------
 airflow/jobs.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/cc62fd06/airflow/jobs.py
----------------------------------------------------------------------
diff --git a/airflow/jobs.py b/airflow/jobs.py
index 106859f..459ff0f 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -1810,7 +1810,8 @@ class SchedulerJob(BaseJob):
             # Also save this task instance to the DB.
             self.log.info("Creating / updating %s in ORM", ti)
             session.merge(ti)
-            session.commit()
+        # commit batch
+        session.commit()
 
         # Record import errors into the ORM
         try: