You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2015/06/10 13:43:55 UTC
[jira] [Comment Edited] (SPARK-8275) HistoryServer caches
incomplete App UIs
[ https://issues.apache.org/jira/browse/SPARK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14579273#comment-14579273 ]
Steve Loughran edited comment on SPARK-8275 at 6/10/15 11:43 AM:
-----------------------------------------------------------------
The cache code is from google; history server provides a method to get the data for an entry, but there's no logic in the cache itself to have a refresh time on entries.
One solution would be
# cache entries to include a timestamp and completed flag alongside the SparkUI instances
# direct all cache.get operations through a single method in HistoryServer
# have that method do something like
{code}
def getUI(id: String): SparkUI = {
var cacheEntry = cache.get(id)
if (!cacheEntry.completed && (cacheEntry.timestamp + expiryTime) < now()) {
cache.release(id)
cache.get(id)
}
cacheEntry
}
{code}
this will leave out of date entries in the cache, but on any retrieval trigger the rebuild.
was (Author: stevel@apache.org):
The cache code is from google; history server provides a method to get the data for an entry, but there's no logic in the cache itself to have a refresh time on entries.
One solution would be
# cache entries to include a timestamp and completed flag alongside the SparkUI instances
# direct all cache.get operations through a single method in HistoryServer
# have that method do something like
{code}
def getUI(id: String): SparkUI = {
var cacheEntry = cache.get(id)
if (!cacheEntry.completed && (cacheEntry.timestamp + expiryTime) < now()) {
cache.release(id)
cache.get(id)
}
cacheEntry
}
this will leave out of date entries in the cache, but on any retrieval trigger the rebuild.
> HistoryServer caches incomplete App UIs
> ---------------------------------------
>
> Key: SPARK-8275
> URL: https://issues.apache.org/jira/browse/SPARK-8275
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.3.1
> Reporter: Steve Loughran
>
> The history server caches applications retrieved from the {{ApplicationHistoryProvider.getAppUI()}} call for performance: it's expensive to rebuild.
> However, this cache also includes incomplete applications, as well as completed ones —and it never attempts to refresh the incomplete application.
> As a result, if you do a GET of the history of a running application, even after the application is finished, you'll still get the web UI/history as it was when that first GET was issued.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org