[jira] [Commented] (AURORA-1847) Eliminate sequential scan in MemTaskStore.getJobKeys()

["Zameer Manji (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AURORA-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726960#comment-15726960 ] 

Zameer Manji commented on AURORA-1847:
--------------------------------------

Could this be resolved by moving to {{DBTaskStore}} or does that have too many drawbacks?

> Eliminate sequential scan in MemTaskStore.getJobKeys()
> ------------------------------------------------------
>
>                 Key: AURORA-1847
>                 URL: https://issues.apache.org/jira/browse/AURORA-1847
>             Project: Aurora
>          Issue Type: Story
>          Components: Efficiency, UI
>            Reporter: Mehrdad Nurolahzade
>            Priority: Minor
>              Labels: newbie
>
> The existing {{TaskStoreBenchmarks}} shows {{DBTaskStore}} is almost two orders of magnitude faster than {{MemTaskStore}} when it comes to {{getJobKeys()}}:
> {code}
> Benchmark                                       (numTasks)   Mode  Cnt      Score      Error  Units
> TaskStoreBenchmarks.DBFetchTasksBenchmark.run        10000  thrpt    5  78430.531 ± 3255.027  ops/s
> TaskStoreBenchmarks.DBFetchTasksBenchmark.run        50000  thrpt    5  50774.988 ± 8986.951  ops/s
> TaskStoreBenchmarks.DBFetchTasksBenchmark.run       100000  thrpt    5   2480.074 ± 9833.122  ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run       10000  thrpt    5   1189.568 ±  108.146  ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run       50000  thrpt    5    124.990 ±   27.605  ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run      100000  thrpt    5     35.724 ±   15.101  ops/s
> {code}
> If scheduler is configured to run with the {{MemTaskStore}} every hit on scheduler page ({{/scheduler}}) causes a call to {{MemTaskStore.getJobKeys()}}. 
> The implementation of this method is currently very inefficient as it results in a sequential scan of the task store and then mapping to their respective job keys. The sequential scan and mapping to job key can be eliminated by simply returning the key set of the existing secondary index  {{job}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)