You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "guoxiaolongzte (JIRA)" <ji...@apache.org> on 2018/02/26 07:11:00 UTC
[jira] [Commented] (SPARK-23492) Application shows up as running in
history server even when latest attempt has completed
[ https://issues.apache.org/jira/browse/SPARK-23492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376458#comment-16376458 ]
guoxiaolongzte commented on SPARK-23492:
----------------------------------------
I tested with the latest environment, but I did not reproduce your problem, I think there is a problem, please email srowen instead of creating a jira. Thanks.
[ {
"id" : "app-20180226150516-0002",
"name" : "Spark shell",
"attempts" : [ {
"startTime" : "2018-02-26T07:05:15.006GMT",
"endTime" : "1969-12-31T23:59:59.999GMT",
"lastUpdated" : "2018-02-26T07:05:19.814GMT",
"duration" : 0,
"sparkUser" : "root",
{color:#FF0000}"completed" : false,{color}
"appSparkVersion" : "2.4.0-SNAPSHOT",
"startTimeEpoch" : 1519628715006,
"endTimeEpoch" : -1,
"lastUpdatedEpoch" : 1519628719814
} ]
}, {
"id" : "app-20180212094839-0008",
"name" : "SparkSQL::10.43.183.120",
"attempts" : [ {
"startTime" : "2018-02-12T01:48:38.165GMT",
"endTime" : "1969-12-31T23:59:59.999GMT",
"lastUpdated" : "2018-02-26T06:58:00.065GMT",
"duration" : 0,
"sparkUser" : "root",
{color:#FF0000}"completed" : false,{color}
"appSparkVersion" : "2.4.0-SNAPSHOT",
"startTimeEpoch" : 1518400118165,
"endTimeEpoch" : -1,
"lastUpdatedEpoch" : 1519628280065
} ]
> Application shows up as running in history server even when latest attempt has completed
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-23492
> URL: https://issues.apache.org/jira/browse/SPARK-23492
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, YARN
> Affects Versions: 2.2.0
> Reporter: Kevin Kim
> Priority: Major
>
> Hi, the Spark history server API "/application?status=running" returns applications whose latest attempts have completed (see example output below). Spark code says that an application is considered "running" when any one of the attempts are not "completed". [https://github.com/apache/spark/blob/1cc34f3e58c92dd06545727e9d931008a1082bbf/core/src/main/scala/org/apache/spark/status/api/v1/ApplicationListResource.scala#L44]
> In my case, attempt 1 is shown as incomplete, but the attempt has finished and is stamped with an endTime. Is this a bug in Spark history server?
> {code:java}
> "attempts" : [ {
> "attemptId" : "2",
> "startTime" : "2018-02-14T23:59:17.785GMT",
> "endTime" : "2018-02-15T08:08:28.927GMT",
> "lastUpdated" : "2018-02-15T08:08:28.949GMT",
> "duration" : 29351142,
> "sparkUser" : {omitted},
> "completed" : true,
> "startTimeEpoch" : 1518652757785,
> "endTimeEpoch" : 1518682108927,
> "lastUpdatedEpoch" : 1518682108949
> }, {
> "attemptId" : "1",
> "startTime" : "2018-02-14T23:53:02.629GMT",
> "endTime" : "2018-02-14T23:59:13.426GMT",
> "lastUpdated" : "2018-02-23T05:03:45.434GMT",
> "duration" : 370797,
> "sparkUser" : {omitted},
> "completed" : false,
> "startTimeEpoch" : 1518652382629,
> "endTimeEpoch" : 1518652753426,
> "lastUpdatedEpoch" : 1519362225434
> } ]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org