[jira] [Created] (AURORA-1674) Migration from MemTaskStore to DbTaskStore can take hours

["Zameer Manji (JIRA)" <ji...@apache.org>]

Zameer Manji created AURORA-1674:
------------------------------------

             Summary: Migration from MemTaskStore to DbTaskStore can take hours
                 Key: AURORA-1674
                 URL: https://issues.apache.org/jira/browse/AURORA-1674
             Project: Aurora
          Issue Type: Bug
            Reporter: Zameer Manji


Migrating an Aurora instance which has about 330k tasks in MemTaskStore to DbTaskStore took over 13h.

This can be replicated as follows:
# Schedule thousands of tasks with a scheduler on MemTaskStore
# Use {{aurora_admin}} to create a snapshot.
# Redeploy the scheduler with DbTaskStore enabled
# Watch as Snapshot recovery and inserting tasks takes several hours

Stack dumps of this process indicates that all of the time was spent in H2 handing the consequences of inserting hundreds of thousands of records.

I wasn't able to figure out what precisely was the root cause but here are some possible suspects
* Generating the SQL for delete tasks which would involve creating a delete statement with 330k task ids.
* H2 transaction/locking configuration making this mass insert very slow.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)