You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by John Lilley <jo...@redpoint.net> on 2013/06/06 20:10:54 UTC
YARN task default output
Do tasks that spawn from a YARN AM have a default working directory? Does stdout/stderr get captured anywhere by default?
I ask because I am setting up tests with the "distributed shell" AM and want to know if basic commands (e.g. ls) will send stdout/stderr somewhere I can see at the end.
Thanks
John
Re: YARN task default output
Posted by Arun C Murthy <ac...@hortonworks.com>.
You can enable log-aggregation where-by your application logs will be uploaded to HDFS.
Overview here:
http://hortonworks.com/blog/apache-hadoop-yarn-nodemanager/
Configurations from http://hadoop.apache.org/docs/r2.0.4-alpha/hadoop-project-dist/hadoop-common/ClusterSetup.html which are relevant are:
yarn.log-aggregation-enable -> true
yarn.nodemanager.remote-app-log-dir -> /logs (directory on hdfs which will need to be created before you start your cluster)
hth,
Arun
On Jun 6, 2013, at 11:10 AM, John Lilley wrote:
> Do tasks that spawn from a YARN AM have a default working directory? Does stdout/stderr get captured anywhere by default?
> I ask because I am setting up tests with the “distributed shell” AM and want to know if basic commands (e.g. ls) will send stdout/stderr somewhere I can see at the end.
> Thanks
> John
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: YARN task default output
Posted by Arun C Murthy <ac...@hortonworks.com>.
You can enable log-aggregation where-by your application logs will be uploaded to HDFS.
Overview here:
http://hortonworks.com/blog/apache-hadoop-yarn-nodemanager/
Configurations from http://hadoop.apache.org/docs/r2.0.4-alpha/hadoop-project-dist/hadoop-common/ClusterSetup.html which are relevant are:
yarn.log-aggregation-enable -> true
yarn.nodemanager.remote-app-log-dir -> /logs (directory on hdfs which will need to be created before you start your cluster)
hth,
Arun
On Jun 6, 2013, at 11:10 AM, John Lilley wrote:
> Do tasks that spawn from a YARN AM have a default working directory? Does stdout/stderr get captured anywhere by default?
> I ask because I am setting up tests with the “distributed shell” AM and want to know if basic commands (e.g. ls) will send stdout/stderr somewhere I can see at the end.
> Thanks
> John
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: YARN task default output
Posted by Arun C Murthy <ac...@hortonworks.com>.
You can enable log-aggregation where-by your application logs will be uploaded to HDFS.
Overview here:
http://hortonworks.com/blog/apache-hadoop-yarn-nodemanager/
Configurations from http://hadoop.apache.org/docs/r2.0.4-alpha/hadoop-project-dist/hadoop-common/ClusterSetup.html which are relevant are:
yarn.log-aggregation-enable -> true
yarn.nodemanager.remote-app-log-dir -> /logs (directory on hdfs which will need to be created before you start your cluster)
hth,
Arun
On Jun 6, 2013, at 11:10 AM, John Lilley wrote:
> Do tasks that spawn from a YARN AM have a default working directory? Does stdout/stderr get captured anywhere by default?
> I ask because I am setting up tests with the “distributed shell” AM and want to know if basic commands (e.g. ls) will send stdout/stderr somewhere I can see at the end.
> Thanks
> John
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: YARN task default output
Posted by Arun C Murthy <ac...@hortonworks.com>.
You can enable log-aggregation where-by your application logs will be uploaded to HDFS.
Overview here:
http://hortonworks.com/blog/apache-hadoop-yarn-nodemanager/
Configurations from http://hadoop.apache.org/docs/r2.0.4-alpha/hadoop-project-dist/hadoop-common/ClusterSetup.html which are relevant are:
yarn.log-aggregation-enable -> true
yarn.nodemanager.remote-app-log-dir -> /logs (directory on hdfs which will need to be created before you start your cluster)
hth,
Arun
On Jun 6, 2013, at 11:10 AM, John Lilley wrote:
> Do tasks that spawn from a YARN AM have a default working directory? Does stdout/stderr get captured anywhere by default?
> I ask because I am setting up tests with the “distributed shell” AM and want to know if basic commands (e.g. ls) will send stdout/stderr somewhere I can see at the end.
> Thanks
> John
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/