[jira] [Assigned] (AURORA-1090) Optimize or remove shard uniqueness check from StorageBackfill

["David McLaughlin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AURORA-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David McLaughlin reassigned AURORA-1090:
----------------------------------------

    Assignee: David McLaughlin

> Optimize or remove shard uniqueness check from StorageBackfill
> --------------------------------------------------------------
>
>                 Key: AURORA-1090
>                 URL: https://issues.apache.org/jira/browse/AURORA-1090
>             Project: Aurora
>          Issue Type: Task
>          Components: Scheduler, Technical Debt
>            Reporter: Bill Farner
>            Assignee: David McLaughlin
>            Priority: Critical
>
> We have noticed that during scheduler startup, the operation, there can be a significant amount of time spent between the following log lines:
> {noformat}
> Performing shard uniqueness sanity check.
> storage state machine transition PREPARED -> READY
> {noformat}
> Looking at what happens in the scheduler between those points, the expensive operation seems to be {{guaranteeShardUniqueness}}.
> This operation aims to validate the integrity of the storage, but its value is dubious.  There are many other things that could be done to validate integrity, but they should probably not be done every time the scheduler loads its database.
> If the operation is kept, it can be dramatically optimized.  It currently performs an O(n^2) scan of tasks, and this could trivially be reduced to O\(n\).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)