Application is "always" in process when I check out logs of completed application

[amghost <zh...@outlook.com>]

I run spark application in spark standalone cluster with client deploy mode.
I want to check out the logs of my finished application, but I always get  a
page telling me "Application history not found - Application xxx is still in
process".
I am pretty sure that the application has indeed completed because I can see
it in the Completed Applications list show by Spark WebUI, and I have also
found the log file with suffix ".inprocess"in the directory set by
"spark.eventLog.dir" in my spark-default.conf

Oh, BTW, I am using spark 1.3.0

So, is there anything I missed?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Application-is-always-in-process-when-I-check-out-logs-of-completed-application-tp23123.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Application is "always" in process when I check out logs of completed application

[Richard Marscher <rm...@localytics.com>]

I had the same issue a couple days ago. It's a bug in 1.3.0 that is fixed
in 1.3.1 and up.

https://issues.apache.org/jira/browse/SPARK-6036

The ordering that the event logs are moved from in-progress to complete is
coded to be after the Master tries to build the history page for the logs.
The only reason it even works on occasion in 1.3.0 is because the Master
part is run asynchronously and the event log status change is synchronous,
so the Master part on some occasions could be executed afterwards as a race
condition.

On Wed, Jun 3, 2015 at 2:17 AM, ayan guha <gu...@gmail.com> wrote:

> Have you done sc.stop() ? :)
> On 3 Jun 2015 14:05, "amghost" <zh...@outlook.com> wrote:
>
>> I run spark application in spark standalone cluster with client deploy
>> mode.
>> I want to check out the logs of my finished application, but I always
>> get  a
>> page telling me "Application history not found - Application xxx is still
>> in
>> process".
>> I am pretty sure that the application has indeed completed because I can
>> see
>> it in the Completed Applications list show by Spark WebUI, and I have also
>> found the log file with suffix ".inprocess"in the directory set by
>> "spark.eventLog.dir" in my spark-default.conf
>>
>> Oh, BTW, I am using spark 1.3.0
>>
>> So, is there anything I missed?
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Application-is-always-in-process-when-I-check-out-logs-of-completed-application-tp23123.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>

Re: Application is "always" in process when I check out logs of completed application

[ayan guha <gu...@gmail.com>]

Have you done sc.stop() ? :)
On 3 Jun 2015 14:05, "amghost" <zh...@outlook.com> wrote:

> I run spark application in spark standalone cluster with client deploy
> mode.
> I want to check out the logs of my finished application, but I always get
> a
> page telling me "Application history not found - Application xxx is still
> in
> process".
> I am pretty sure that the application has indeed completed because I can
> see
> it in the Completed Applications list show by Spark WebUI, and I have also
> found the log file with suffix ".inprocess"in the directory set by
> "spark.eventLog.dir" in my spark-default.conf
>
> Oh, BTW, I am using spark 1.3.0
>
> So, is there anything I missed?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Application-is-always-in-process-when-I-check-out-logs-of-completed-application-tp23123.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>