You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Rachana Srivastava <Ra...@markmonitor.com> on 2016/07/19 22:23:20 UTC
Missing Exector Logs From Yarn After Spark Failure
I am trying to find the root cause of recent Spark application failure in production. When the Spark application is running I can check NodeManager's yarn.nodemanager.log-dir property to get the Spark executor container logs.
The container has logs for both the running Spark applications
Here is the view of the container logs: drwx--x--- 3 yarn yarn 51 Jul 19 09:04 application_1467068598418_0209 drwx--x--- 5 yarn yarn 141 Jul 19 09:04 application_1467068598418_0210
But when the application is killed both the application logs are automatically deleted. I have set all the log retention setting etc in Yarn to a very large number. But still these logs are deleted as soon as the Spark applications are crashed.
Question: How can we retain these Spark application logs in Yarn for debugging when the Spark application is crashed for some reason.
Re: Missing Exector Logs From Yarn After Spark Failure
Posted by ayan guha <gu...@gmail.com>.
If YARN log aggregation is enabled then logs will be moved to HDFS. You can
use yarn logs -applicationId <applicationId> to view those logs.
On Wed, Jul 20, 2016 at 8:58 AM, Ted Yu <yu...@gmail.com> wrote:
> What's the value for yarn.log-aggregation.retain-seconds
> and yarn.log-aggregation-enable ?
>
> Which hadoop release are you using ?
>
> Thanks
>
> On Tue, Jul 19, 2016 at 3:23 PM, Rachana Srivastava <
> Rachana.Srivastava@markmonitor.com> wrote:
>
>> I am trying to find the root cause of recent Spark application failure in
>> production. When the Spark application is running I can check NodeManager's
>> yarn.nodemanager.log-dir property to get the Spark executor container logs.
>>
>> The container has logs for both the running Spark applications
>>
>> Here is the view of the container logs: drwx--x--- 3 yarn yarn 51 Jul 19
>> 09:04 application_1467068598418_0209 drwx--x--- 5 yarn yarn 141 Jul 19
>> 09:04 application_1467068598418_0210
>>
>> But when the application is killed both the application logs are
>> automatically deleted. I have set all the log retention setting etc in Yarn
>> to a very large number. But still these logs are deleted as soon as the
>> Spark applications are crashed.
>>
>> Question: How can we retain these Spark application logs in Yarn for
>> debugging when the Spark application is crashed for some reason.
>>
>
>
--
Best Regards,
Ayan Guha
Re: Missing Exector Logs From Yarn After Spark Failure
Posted by Ted Yu <yu...@gmail.com>.
What's the value for yarn.log-aggregation.retain-seconds
and yarn.log-aggregation-enable ?
Which hadoop release are you using ?
Thanks
On Tue, Jul 19, 2016 at 3:23 PM, Rachana Srivastava <
Rachana.Srivastava@markmonitor.com> wrote:
> I am trying to find the root cause of recent Spark application failure in
> production. When the Spark application is running I can check NodeManager's
> yarn.nodemanager.log-dir property to get the Spark executor container logs.
>
> The container has logs for both the running Spark applications
>
> Here is the view of the container logs: drwx--x--- 3 yarn yarn 51 Jul 19
> 09:04 application_1467068598418_0209 drwx--x--- 5 yarn yarn 141 Jul 19
> 09:04 application_1467068598418_0210
>
> But when the application is killed both the application logs are
> automatically deleted. I have set all the log retention setting etc in Yarn
> to a very large number. But still these logs are deleted as soon as the
> Spark applications are crashed.
>
> Question: How can we retain these Spark application logs in Yarn for
> debugging when the Spark application is crashed for some reason.
>