You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Hemanth Yamijala <yh...@thoughtworks.com> on 2012/09/10 12:49:04 UTC
Log aggregation in Yarn
Hi,
I enabled log aggregation in Yarn and can see that files are getting
created under the configured directory on HDFS. I can access the files via
FS shell. However, when I try to retrieve the logs via the history server,
they fail with a message:
Logs not available for attempt_1347261424213_0001_r_000000_0. Aggregation
may not be complete, Check back later or try the nodemanager.
This is with trunk. From code, it looks like this happens because the
service is unable to access the file on HDFS. However, this is a single
node setup, all processes running as the same user. Security is off in
HDFS. Anything else wrong ? Do I need to configure something more ?
Thanks
Hemanth
Re: Log aggregation in Yarn
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
I figured out what the issue is.
I'd configured my yarn.nodemanager.remote-app-log-dir to be
${yarn.log.dir}/logs. This doesn't per se make sense because this is a
remote location (on HDFS) and not related to yarn.log.dir which is more
like a local directory. However, the issue was that the script which starts
the history server ultimately uses the 'mapred' script that doesn't export
yarn.log.dir as a system property. Hence, while the logs were getting
aggregated correctly, the history server wasn't able to resolve to the
right path to show the logs from.
Fixing the value of the yarn.nodemanager.remote-app-log-dir variable fixed
the issue for me.
Thanks
Hemanth
On Mon, Sep 10, 2012 at 4:52 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Hemanth,
>
> I am still getting my hands dirty on yarn, so this is preliminary –
> maybe as the hdfs path in AggregatedLogsBlock points to /tmp/logs and you
> say service is unable to read it, possibly check perm or change the
> configuration in yarn-site.xml and try?
>
> Thanks
> Rekha
>
> From: Hemanth Yamijala <yh...@thoughtworks.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Mon, 10 Sep 2012 16:19:04 +0530
> To: <us...@hadoop.apache.org>
> Subject: Log aggregation in Yarn
>
> Hi,
>
> I enabled log aggregation in Yarn and can see that files are getting
> created under the configured directory on HDFS. I can access the files via
> FS shell. However, when I try to retrieve the logs via the history server,
> they fail with a message:
>
> Logs not available for attempt_1347261424213_0001_r_000000_0.
> Aggregation may not be complete, Check back later or try the nodemanager.
>
> This is with trunk. From code, it looks like this happens because the
> service is unable to access the file on HDFS. However, this is a single
> node setup, all processes running as the same user. Security is off in
> HDFS. Anything else wrong ? Do I need to configure something more ?
>
> Thanks
> Hemanth
>
Re: Log aggregation in Yarn
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
I figured out what the issue is.
I'd configured my yarn.nodemanager.remote-app-log-dir to be
${yarn.log.dir}/logs. This doesn't per se make sense because this is a
remote location (on HDFS) and not related to yarn.log.dir which is more
like a local directory. However, the issue was that the script which starts
the history server ultimately uses the 'mapred' script that doesn't export
yarn.log.dir as a system property. Hence, while the logs were getting
aggregated correctly, the history server wasn't able to resolve to the
right path to show the logs from.
Fixing the value of the yarn.nodemanager.remote-app-log-dir variable fixed
the issue for me.
Thanks
Hemanth
On Mon, Sep 10, 2012 at 4:52 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Hemanth,
>
> I am still getting my hands dirty on yarn, so this is preliminary –
> maybe as the hdfs path in AggregatedLogsBlock points to /tmp/logs and you
> say service is unable to read it, possibly check perm or change the
> configuration in yarn-site.xml and try?
>
> Thanks
> Rekha
>
> From: Hemanth Yamijala <yh...@thoughtworks.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Mon, 10 Sep 2012 16:19:04 +0530
> To: <us...@hadoop.apache.org>
> Subject: Log aggregation in Yarn
>
> Hi,
>
> I enabled log aggregation in Yarn and can see that files are getting
> created under the configured directory on HDFS. I can access the files via
> FS shell. However, when I try to retrieve the logs via the history server,
> they fail with a message:
>
> Logs not available for attempt_1347261424213_0001_r_000000_0.
> Aggregation may not be complete, Check back later or try the nodemanager.
>
> This is with trunk. From code, it looks like this happens because the
> service is unable to access the file on HDFS. However, this is a single
> node setup, all processes running as the same user. Security is off in
> HDFS. Anything else wrong ? Do I need to configure something more ?
>
> Thanks
> Hemanth
>
Re: Log aggregation in Yarn
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
I figured out what the issue is.
I'd configured my yarn.nodemanager.remote-app-log-dir to be
${yarn.log.dir}/logs. This doesn't per se make sense because this is a
remote location (on HDFS) and not related to yarn.log.dir which is more
like a local directory. However, the issue was that the script which starts
the history server ultimately uses the 'mapred' script that doesn't export
yarn.log.dir as a system property. Hence, while the logs were getting
aggregated correctly, the history server wasn't able to resolve to the
right path to show the logs from.
Fixing the value of the yarn.nodemanager.remote-app-log-dir variable fixed
the issue for me.
Thanks
Hemanth
On Mon, Sep 10, 2012 at 4:52 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Hemanth,
>
> I am still getting my hands dirty on yarn, so this is preliminary –
> maybe as the hdfs path in AggregatedLogsBlock points to /tmp/logs and you
> say service is unable to read it, possibly check perm or change the
> configuration in yarn-site.xml and try?
>
> Thanks
> Rekha
>
> From: Hemanth Yamijala <yh...@thoughtworks.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Mon, 10 Sep 2012 16:19:04 +0530
> To: <us...@hadoop.apache.org>
> Subject: Log aggregation in Yarn
>
> Hi,
>
> I enabled log aggregation in Yarn and can see that files are getting
> created under the configured directory on HDFS. I can access the files via
> FS shell. However, when I try to retrieve the logs via the history server,
> they fail with a message:
>
> Logs not available for attempt_1347261424213_0001_r_000000_0.
> Aggregation may not be complete, Check back later or try the nodemanager.
>
> This is with trunk. From code, it looks like this happens because the
> service is unable to access the file on HDFS. However, this is a single
> node setup, all processes running as the same user. Security is off in
> HDFS. Anything else wrong ? Do I need to configure something more ?
>
> Thanks
> Hemanth
>
Re: Log aggregation in Yarn
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
I figured out what the issue is.
I'd configured my yarn.nodemanager.remote-app-log-dir to be
${yarn.log.dir}/logs. This doesn't per se make sense because this is a
remote location (on HDFS) and not related to yarn.log.dir which is more
like a local directory. However, the issue was that the script which starts
the history server ultimately uses the 'mapred' script that doesn't export
yarn.log.dir as a system property. Hence, while the logs were getting
aggregated correctly, the history server wasn't able to resolve to the
right path to show the logs from.
Fixing the value of the yarn.nodemanager.remote-app-log-dir variable fixed
the issue for me.
Thanks
Hemanth
On Mon, Sep 10, 2012 at 4:52 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Hemanth,
>
> I am still getting my hands dirty on yarn, so this is preliminary –
> maybe as the hdfs path in AggregatedLogsBlock points to /tmp/logs and you
> say service is unable to read it, possibly check perm or change the
> configuration in yarn-site.xml and try?
>
> Thanks
> Rekha
>
> From: Hemanth Yamijala <yh...@thoughtworks.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Mon, 10 Sep 2012 16:19:04 +0530
> To: <us...@hadoop.apache.org>
> Subject: Log aggregation in Yarn
>
> Hi,
>
> I enabled log aggregation in Yarn and can see that files are getting
> created under the configured directory on HDFS. I can access the files via
> FS shell. However, when I try to retrieve the logs via the history server,
> they fail with a message:
>
> Logs not available for attempt_1347261424213_0001_r_000000_0.
> Aggregation may not be complete, Check back later or try the nodemanager.
>
> This is with trunk. From code, it looks like this happens because the
> service is unable to access the file on HDFS. However, this is a single
> node setup, all processes running as the same user. Security is off in
> HDFS. Anything else wrong ? Do I need to configure something more ?
>
> Thanks
> Hemanth
>
Re: Log aggregation in Yarn
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Hemanth,
I am still getting my hands dirty on yarn, so this is preliminary – maybe as the hdfs path in AggregatedLogsBlock points to /tmp/logs and you say service is unable to read it, possibly check perm or change the configuration in yarn-site.xml and try?
Thanks
Rekha
From: Hemanth Yamijala <yh...@thoughtworks.com>>
Reply-To: <us...@hadoop.apache.org>>
Date: Mon, 10 Sep 2012 16:19:04 +0530
To: <us...@hadoop.apache.org>>
Subject: Log aggregation in Yarn
Hi,
I enabled log aggregation in Yarn and can see that files are getting created under the configured directory on HDFS. I can access the files via FS shell. However, when I try to retrieve the logs via the history server, they fail with a message:
Logs not available for attempt_1347261424213_0001_r_000000_0. Aggregation may not be complete, Check back later or try the nodemanager.
This is with trunk. From code, it looks like this happens because the service is unable to access the file on HDFS. However, this is a single node setup, all processes running as the same user. Security is off in HDFS. Anything else wrong ? Do I need to configure something more ?
Thanks
Hemanth
Re: Log aggregation in Yarn
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Hemanth,
I am still getting my hands dirty on yarn, so this is preliminary – maybe as the hdfs path in AggregatedLogsBlock points to /tmp/logs and you say service is unable to read it, possibly check perm or change the configuration in yarn-site.xml and try?
Thanks
Rekha
From: Hemanth Yamijala <yh...@thoughtworks.com>>
Reply-To: <us...@hadoop.apache.org>>
Date: Mon, 10 Sep 2012 16:19:04 +0530
To: <us...@hadoop.apache.org>>
Subject: Log aggregation in Yarn
Hi,
I enabled log aggregation in Yarn and can see that files are getting created under the configured directory on HDFS. I can access the files via FS shell. However, when I try to retrieve the logs via the history server, they fail with a message:
Logs not available for attempt_1347261424213_0001_r_000000_0. Aggregation may not be complete, Check back later or try the nodemanager.
This is with trunk. From code, it looks like this happens because the service is unable to access the file on HDFS. However, this is a single node setup, all processes running as the same user. Security is off in HDFS. Anything else wrong ? Do I need to configure something more ?
Thanks
Hemanth
Re: Log aggregation in Yarn
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Hemanth,
I am still getting my hands dirty on yarn, so this is preliminary – maybe as the hdfs path in AggregatedLogsBlock points to /tmp/logs and you say service is unable to read it, possibly check perm or change the configuration in yarn-site.xml and try?
Thanks
Rekha
From: Hemanth Yamijala <yh...@thoughtworks.com>>
Reply-To: <us...@hadoop.apache.org>>
Date: Mon, 10 Sep 2012 16:19:04 +0530
To: <us...@hadoop.apache.org>>
Subject: Log aggregation in Yarn
Hi,
I enabled log aggregation in Yarn and can see that files are getting created under the configured directory on HDFS. I can access the files via FS shell. However, when I try to retrieve the logs via the history server, they fail with a message:
Logs not available for attempt_1347261424213_0001_r_000000_0. Aggregation may not be complete, Check back later or try the nodemanager.
This is with trunk. From code, it looks like this happens because the service is unable to access the file on HDFS. However, this is a single node setup, all processes running as the same user. Security is off in HDFS. Anything else wrong ? Do I need to configure something more ?
Thanks
Hemanth
Re: Log aggregation in Yarn
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Hemanth,
I am still getting my hands dirty on yarn, so this is preliminary – maybe as the hdfs path in AggregatedLogsBlock points to /tmp/logs and you say service is unable to read it, possibly check perm or change the configuration in yarn-site.xml and try?
Thanks
Rekha
From: Hemanth Yamijala <yh...@thoughtworks.com>>
Reply-To: <us...@hadoop.apache.org>>
Date: Mon, 10 Sep 2012 16:19:04 +0530
To: <us...@hadoop.apache.org>>
Subject: Log aggregation in Yarn
Hi,
I enabled log aggregation in Yarn and can see that files are getting created under the configured directory on HDFS. I can access the files via FS shell. However, when I try to retrieve the logs via the history server, they fail with a message:
Logs not available for attempt_1347261424213_0001_r_000000_0. Aggregation may not be complete, Check back later or try the nodemanager.
This is with trunk. From code, it looks like this happens because the service is unable to access the file on HDFS. However, this is a single node setup, all processes running as the same user. Security is off in HDFS. Anything else wrong ? Do I need to configure something more ?
Thanks
Hemanth