[jira] [Resolved] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

["Andrew Or (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Or resolved SPARK-17672.
-------------------------------
          Resolution: Fixed
       Fix Version/s: 2.0.1
    Target Version/s: 2.0.1

> Spark 2.0 history server web Ui takes too long for a single application
> -----------------------------------------------------------------------
>
>                 Key: SPARK-17672
>                 URL: https://issues.apache.org/jira/browse/SPARK-17672
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 2.0.0
>            Reporter: Gang Wu
>             Fix For: 2.0.1
>
>
> When there are 10K application history in the history server back end, it can take a very long time to even get a single application history page. After some investigation, I found the root cause was the following piece of code: 
> {code:title=OneApplicationResource.scala|borderStyle=solid}
> @Produces(Array(MediaType.APPLICATION_JSON))
> private[v1] class OneApplicationResource(uiRoot: UIRoot) {
>   @GET
>   def getApp(@PathParam("appId") appId: String): ApplicationInfo = {
>     val apps = uiRoot.getApplicationInfoList.find { _.id == appId }
>     apps.getOrElse(throw new NotFoundException("unknown app: " + appId))
>   }
> }
> {code}
> Although all application history infos are stored in a LinkedHashMap, here to code transforms the map to an iterator and then uses the find() api which is O( n) instead of O(1) from a map.get() operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org