You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by "Brian C. Huffman" <bh...@etinternational.com> on 2014/07/15 16:49:00 UTC
YARN Logs
All,
I am running a small cluster with hadoop-2.2.0 installed on an NFS
shared directory. Since all nodes can access, I do not want to enable
log aggregation.
My understanding was that if aggregation wasn't enabled, the 'yarn logs'
command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
isn't happening:
[hadoop@host1 ~]$ yarn logs -applicationId application_1405396841766_0003
14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
host1/172.17.1.1:8032
Logs not available at /tmp/logs/hadoop/logs/application_1405396841766_0003
Log aggregation has not completed or is not enabled.
Can anyone suggest what I might be doing wrong?
Thanks,
Brian
Re: YARN Logs
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Vinod,
When you say "it's not easy, but possible", what would have to be done
to make it work?
Thanks,
Brian
On 07/15/2014 11:23 PM, Vinod Kumar Vavilapalli wrote:
> Adam is right.
>
> "yarn logs" command only works when log-aggregation is enabled. It's
> not easy but possible to make it work when aggregation is disabled.
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
> On Tue, Jul 15, 2014 at 10:03 AM, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Looking at the docs, the yarn.nodemanager.remote-app-log-dir
> variable description says "Where to aggregate logs to." But I
> don't want to use log aggregation.
>
> Does the 'yarn logs' command look anywhere else if
> yarn.log-aggregation-enable is set to false?
>
> -b
>
>
> On 07/15/2014 12:10 PM, Adam Kawa wrote:
>> For example, the remote location is configured
>> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
>> This is why, you see:
>> /Logs not available at
>> */tmp/logs*/hadoop/logs/application_1405396841766_0003./
>>
>> PS.
>> The full path is configured via
>> ${yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>>
>>
>>
>> 2014-07-15 18:08 GMT+02:00 Adam Kawa <kawa.adam@gmail.com
>> <ma...@gmail.com>>:
>>
>> IMHO,
>> $ yarn logs looks for aggregated logs at remote location.
>>
>>
>>
>> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman
>> <bhuffman@etinternational.com
>> <ma...@etinternational.com>>:
>>
>> All,
>>
>> I am running a small cluster with hadoop-2.2.0 installed
>> on an NFS shared directory. Since all nodes can access,
>> I do not want to enable log aggregation.
>>
>> My understanding was that if aggregation wasn't enabled,
>> the 'yarn logs' command would just look in the
>> $HADOOP_HOME/logs/userlogs dir, but that isn't happening:
>> [hadoop@host1 ~]$ yarn logs -applicationId
>> application_1405396841766_0003
>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to
>> ResourceManager at host1/172.17.1.1:8032
>> <http://172.17.1.1:8032>
>> Logs not available at
>> /tmp/logs/hadoop/logs/application_1405396841766_0003
>> Log aggregation has not completed or is not enabled.
>>
>> Can anyone suggest what I might be doing wrong?
>>
>> Thanks,
>> Brian
>>
>>
>>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
> entity to which it is addressed and may contain information that is
> confidential, privileged and exempt from disclosure under applicable
> law. If the reader of this message is not the intended recipient, you
> are hereby notified that any printing, copying, dissemination,
> distribution, disclosure or forwarding of this communication is
> strictly prohibited. If you have received this communication in error,
> please contact the sender immediately and delete it from your system.
> Thank You.
Re: YARN Logs
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Vinod,
When you say "it's not easy, but possible", what would have to be done
to make it work?
Thanks,
Brian
On 07/15/2014 11:23 PM, Vinod Kumar Vavilapalli wrote:
> Adam is right.
>
> "yarn logs" command only works when log-aggregation is enabled. It's
> not easy but possible to make it work when aggregation is disabled.
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
> On Tue, Jul 15, 2014 at 10:03 AM, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Looking at the docs, the yarn.nodemanager.remote-app-log-dir
> variable description says "Where to aggregate logs to." But I
> don't want to use log aggregation.
>
> Does the 'yarn logs' command look anywhere else if
> yarn.log-aggregation-enable is set to false?
>
> -b
>
>
> On 07/15/2014 12:10 PM, Adam Kawa wrote:
>> For example, the remote location is configured
>> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
>> This is why, you see:
>> /Logs not available at
>> */tmp/logs*/hadoop/logs/application_1405396841766_0003./
>>
>> PS.
>> The full path is configured via
>> ${yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>>
>>
>>
>> 2014-07-15 18:08 GMT+02:00 Adam Kawa <kawa.adam@gmail.com
>> <ma...@gmail.com>>:
>>
>> IMHO,
>> $ yarn logs looks for aggregated logs at remote location.
>>
>>
>>
>> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman
>> <bhuffman@etinternational.com
>> <ma...@etinternational.com>>:
>>
>> All,
>>
>> I am running a small cluster with hadoop-2.2.0 installed
>> on an NFS shared directory. Since all nodes can access,
>> I do not want to enable log aggregation.
>>
>> My understanding was that if aggregation wasn't enabled,
>> the 'yarn logs' command would just look in the
>> $HADOOP_HOME/logs/userlogs dir, but that isn't happening:
>> [hadoop@host1 ~]$ yarn logs -applicationId
>> application_1405396841766_0003
>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to
>> ResourceManager at host1/172.17.1.1:8032
>> <http://172.17.1.1:8032>
>> Logs not available at
>> /tmp/logs/hadoop/logs/application_1405396841766_0003
>> Log aggregation has not completed or is not enabled.
>>
>> Can anyone suggest what I might be doing wrong?
>>
>> Thanks,
>> Brian
>>
>>
>>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
> entity to which it is addressed and may contain information that is
> confidential, privileged and exempt from disclosure under applicable
> law. If the reader of this message is not the intended recipient, you
> are hereby notified that any printing, copying, dissemination,
> distribution, disclosure or forwarding of this communication is
> strictly prohibited. If you have received this communication in error,
> please contact the sender immediately and delete it from your system.
> Thank You.
Re: YARN Logs
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Vinod,
When you say "it's not easy, but possible", what would have to be done
to make it work?
Thanks,
Brian
On 07/15/2014 11:23 PM, Vinod Kumar Vavilapalli wrote:
> Adam is right.
>
> "yarn logs" command only works when log-aggregation is enabled. It's
> not easy but possible to make it work when aggregation is disabled.
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
> On Tue, Jul 15, 2014 at 10:03 AM, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Looking at the docs, the yarn.nodemanager.remote-app-log-dir
> variable description says "Where to aggregate logs to." But I
> don't want to use log aggregation.
>
> Does the 'yarn logs' command look anywhere else if
> yarn.log-aggregation-enable is set to false?
>
> -b
>
>
> On 07/15/2014 12:10 PM, Adam Kawa wrote:
>> For example, the remote location is configured
>> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
>> This is why, you see:
>> /Logs not available at
>> */tmp/logs*/hadoop/logs/application_1405396841766_0003./
>>
>> PS.
>> The full path is configured via
>> ${yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>>
>>
>>
>> 2014-07-15 18:08 GMT+02:00 Adam Kawa <kawa.adam@gmail.com
>> <ma...@gmail.com>>:
>>
>> IMHO,
>> $ yarn logs looks for aggregated logs at remote location.
>>
>>
>>
>> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman
>> <bhuffman@etinternational.com
>> <ma...@etinternational.com>>:
>>
>> All,
>>
>> I am running a small cluster with hadoop-2.2.0 installed
>> on an NFS shared directory. Since all nodes can access,
>> I do not want to enable log aggregation.
>>
>> My understanding was that if aggregation wasn't enabled,
>> the 'yarn logs' command would just look in the
>> $HADOOP_HOME/logs/userlogs dir, but that isn't happening:
>> [hadoop@host1 ~]$ yarn logs -applicationId
>> application_1405396841766_0003
>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to
>> ResourceManager at host1/172.17.1.1:8032
>> <http://172.17.1.1:8032>
>> Logs not available at
>> /tmp/logs/hadoop/logs/application_1405396841766_0003
>> Log aggregation has not completed or is not enabled.
>>
>> Can anyone suggest what I might be doing wrong?
>>
>> Thanks,
>> Brian
>>
>>
>>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
> entity to which it is addressed and may contain information that is
> confidential, privileged and exempt from disclosure under applicable
> law. If the reader of this message is not the intended recipient, you
> are hereby notified that any printing, copying, dissemination,
> distribution, disclosure or forwarding of this communication is
> strictly prohibited. If you have received this communication in error,
> please contact the sender immediately and delete it from your system.
> Thank You.
Re: YARN Logs
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Vinod,
When you say "it's not easy, but possible", what would have to be done
to make it work?
Thanks,
Brian
On 07/15/2014 11:23 PM, Vinod Kumar Vavilapalli wrote:
> Adam is right.
>
> "yarn logs" command only works when log-aggregation is enabled. It's
> not easy but possible to make it work when aggregation is disabled.
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
> On Tue, Jul 15, 2014 at 10:03 AM, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Looking at the docs, the yarn.nodemanager.remote-app-log-dir
> variable description says "Where to aggregate logs to." But I
> don't want to use log aggregation.
>
> Does the 'yarn logs' command look anywhere else if
> yarn.log-aggregation-enable is set to false?
>
> -b
>
>
> On 07/15/2014 12:10 PM, Adam Kawa wrote:
>> For example, the remote location is configured
>> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
>> This is why, you see:
>> /Logs not available at
>> */tmp/logs*/hadoop/logs/application_1405396841766_0003./
>>
>> PS.
>> The full path is configured via
>> ${yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>>
>>
>>
>> 2014-07-15 18:08 GMT+02:00 Adam Kawa <kawa.adam@gmail.com
>> <ma...@gmail.com>>:
>>
>> IMHO,
>> $ yarn logs looks for aggregated logs at remote location.
>>
>>
>>
>> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman
>> <bhuffman@etinternational.com
>> <ma...@etinternational.com>>:
>>
>> All,
>>
>> I am running a small cluster with hadoop-2.2.0 installed
>> on an NFS shared directory. Since all nodes can access,
>> I do not want to enable log aggregation.
>>
>> My understanding was that if aggregation wasn't enabled,
>> the 'yarn logs' command would just look in the
>> $HADOOP_HOME/logs/userlogs dir, but that isn't happening:
>> [hadoop@host1 ~]$ yarn logs -applicationId
>> application_1405396841766_0003
>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to
>> ResourceManager at host1/172.17.1.1:8032
>> <http://172.17.1.1:8032>
>> Logs not available at
>> /tmp/logs/hadoop/logs/application_1405396841766_0003
>> Log aggregation has not completed or is not enabled.
>>
>> Can anyone suggest what I might be doing wrong?
>>
>> Thanks,
>> Brian
>>
>>
>>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
> entity to which it is addressed and may contain information that is
> confidential, privileged and exempt from disclosure under applicable
> law. If the reader of this message is not the intended recipient, you
> are hereby notified that any printing, copying, dissemination,
> distribution, disclosure or forwarding of this communication is
> strictly prohibited. If you have received this communication in error,
> please contact the sender immediately and delete it from your system.
> Thank You.
Re: YARN Logs
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Adam is right.
"yarn logs" command only works when log-aggregation is enabled. It's not
easy but possible to make it work when aggregation is disabled.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Tue, Jul 15, 2014 at 10:03 AM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Looking at the docs, the yarn.nodemanager.remote-app-log-dir variable
> description says "Where to aggregate logs to." But I don't want to use log
> aggregation.
>
> Does the 'yarn logs' command look anywhere else if
> yarn.log-aggregation-enable is set to false?
>
> -b
>
>
> On 07/15/2014 12:10 PM, Adam Kawa wrote:
>
> For example, the remote location is configured
> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
> This is why, you see:
> *Logs not available at
> /tmp/logs/hadoop/logs/application_1405396841766_0003.*
>
> PS.
> The full path is configured via $
> {yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>
>
>
> 2014-07-15 18:08 GMT+02:00 Adam Kawa <ka...@gmail.com>:
>
>> IMHO,
>> $ yarn logs looks for aggregated logs at remote location.
>>
>>
>>
>> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bhuffman@etinternational.com
>> >:
>>
>> All,
>>>
>>> I am running a small cluster with hadoop-2.2.0 installed on an NFS
>>> shared directory. Since all nodes can access, I do not want to enable log
>>> aggregation.
>>>
>>> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
>>> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
>>> isn't happening:
>>> [hadoop@host1 ~]$ yarn logs -applicationId
>>> application_1405396841766_0003
>>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
>>> host1/172.17.1.1:8032
>>> Logs not available at
>>> /tmp/logs/hadoop/logs/application_1405396841766_0003
>>> Log aggregation has not completed or is not enabled.
>>>
>>> Can anyone suggest what I might be doing wrong?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: YARN Logs
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Adam is right.
"yarn logs" command only works when log-aggregation is enabled. It's not
easy but possible to make it work when aggregation is disabled.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Tue, Jul 15, 2014 at 10:03 AM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Looking at the docs, the yarn.nodemanager.remote-app-log-dir variable
> description says "Where to aggregate logs to." But I don't want to use log
> aggregation.
>
> Does the 'yarn logs' command look anywhere else if
> yarn.log-aggregation-enable is set to false?
>
> -b
>
>
> On 07/15/2014 12:10 PM, Adam Kawa wrote:
>
> For example, the remote location is configured
> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
> This is why, you see:
> *Logs not available at
> /tmp/logs/hadoop/logs/application_1405396841766_0003.*
>
> PS.
> The full path is configured via $
> {yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>
>
>
> 2014-07-15 18:08 GMT+02:00 Adam Kawa <ka...@gmail.com>:
>
>> IMHO,
>> $ yarn logs looks for aggregated logs at remote location.
>>
>>
>>
>> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bhuffman@etinternational.com
>> >:
>>
>> All,
>>>
>>> I am running a small cluster with hadoop-2.2.0 installed on an NFS
>>> shared directory. Since all nodes can access, I do not want to enable log
>>> aggregation.
>>>
>>> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
>>> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
>>> isn't happening:
>>> [hadoop@host1 ~]$ yarn logs -applicationId
>>> application_1405396841766_0003
>>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
>>> host1/172.17.1.1:8032
>>> Logs not available at
>>> /tmp/logs/hadoop/logs/application_1405396841766_0003
>>> Log aggregation has not completed or is not enabled.
>>>
>>> Can anyone suggest what I might be doing wrong?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: YARN Logs
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Adam is right.
"yarn logs" command only works when log-aggregation is enabled. It's not
easy but possible to make it work when aggregation is disabled.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Tue, Jul 15, 2014 at 10:03 AM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Looking at the docs, the yarn.nodemanager.remote-app-log-dir variable
> description says "Where to aggregate logs to." But I don't want to use log
> aggregation.
>
> Does the 'yarn logs' command look anywhere else if
> yarn.log-aggregation-enable is set to false?
>
> -b
>
>
> On 07/15/2014 12:10 PM, Adam Kawa wrote:
>
> For example, the remote location is configured
> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
> This is why, you see:
> *Logs not available at
> /tmp/logs/hadoop/logs/application_1405396841766_0003.*
>
> PS.
> The full path is configured via $
> {yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>
>
>
> 2014-07-15 18:08 GMT+02:00 Adam Kawa <ka...@gmail.com>:
>
>> IMHO,
>> $ yarn logs looks for aggregated logs at remote location.
>>
>>
>>
>> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bhuffman@etinternational.com
>> >:
>>
>> All,
>>>
>>> I am running a small cluster with hadoop-2.2.0 installed on an NFS
>>> shared directory. Since all nodes can access, I do not want to enable log
>>> aggregation.
>>>
>>> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
>>> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
>>> isn't happening:
>>> [hadoop@host1 ~]$ yarn logs -applicationId
>>> application_1405396841766_0003
>>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
>>> host1/172.17.1.1:8032
>>> Logs not available at
>>> /tmp/logs/hadoop/logs/application_1405396841766_0003
>>> Log aggregation has not completed or is not enabled.
>>>
>>> Can anyone suggest what I might be doing wrong?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: YARN Logs
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Adam is right.
"yarn logs" command only works when log-aggregation is enabled. It's not
easy but possible to make it work when aggregation is disabled.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Tue, Jul 15, 2014 at 10:03 AM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Looking at the docs, the yarn.nodemanager.remote-app-log-dir variable
> description says "Where to aggregate logs to." But I don't want to use log
> aggregation.
>
> Does the 'yarn logs' command look anywhere else if
> yarn.log-aggregation-enable is set to false?
>
> -b
>
>
> On 07/15/2014 12:10 PM, Adam Kawa wrote:
>
> For example, the remote location is configured
> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
> This is why, you see:
> *Logs not available at
> /tmp/logs/hadoop/logs/application_1405396841766_0003.*
>
> PS.
> The full path is configured via $
> {yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>
>
>
> 2014-07-15 18:08 GMT+02:00 Adam Kawa <ka...@gmail.com>:
>
>> IMHO,
>> $ yarn logs looks for aggregated logs at remote location.
>>
>>
>>
>> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bhuffman@etinternational.com
>> >:
>>
>> All,
>>>
>>> I am running a small cluster with hadoop-2.2.0 installed on an NFS
>>> shared directory. Since all nodes can access, I do not want to enable log
>>> aggregation.
>>>
>>> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
>>> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
>>> isn't happening:
>>> [hadoop@host1 ~]$ yarn logs -applicationId
>>> application_1405396841766_0003
>>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
>>> host1/172.17.1.1:8032
>>> Logs not available at
>>> /tmp/logs/hadoop/logs/application_1405396841766_0003
>>> Log aggregation has not completed or is not enabled.
>>>
>>> Can anyone suggest what I might be doing wrong?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: YARN Logs
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Looking at the docs, the yarn.nodemanager.remote-app-log-dir variable
description says "Where to aggregate logs to." But I don't want to use
log aggregation.
Does the 'yarn logs' command look anywhere else if
yarn.log-aggregation-enable is set to false?
-b
On 07/15/2014 12:10 PM, Adam Kawa wrote:
> For example, the remote location is configured
> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
> This is why, you see:
> /Logs not available at
> */tmp/logs*/hadoop/logs/application_1405396841766_0003./
>
> PS.
> The full path is configured via
> ${yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>
>
>
> 2014-07-15 18:08 GMT+02:00 Adam Kawa <kawa.adam@gmail.com
> <ma...@gmail.com>>:
>
> IMHO,
> $ yarn logs looks for aggregated logs at remote location.
>
>
>
> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>:
>
> All,
>
> I am running a small cluster with hadoop-2.2.0 installed on an
> NFS shared directory. Since all nodes can access, I do not
> want to enable log aggregation.
>
> My understanding was that if aggregation wasn't enabled, the
> 'yarn logs' command would just look in the
> $HADOOP_HOME/logs/userlogs dir, but that isn't happening:
> [hadoop@host1 ~]$ yarn logs -applicationId
> application_1405396841766_0003
> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to
> ResourceManager at host1/172.17.1.1:8032 <http://172.17.1.1:8032>
> Logs not available at
> /tmp/logs/hadoop/logs/application_1405396841766_0003
> Log aggregation has not completed or is not enabled.
>
> Can anyone suggest what I might be doing wrong?
>
> Thanks,
> Brian
>
>
>
Re: YARN Logs
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Looking at the docs, the yarn.nodemanager.remote-app-log-dir variable
description says "Where to aggregate logs to." But I don't want to use
log aggregation.
Does the 'yarn logs' command look anywhere else if
yarn.log-aggregation-enable is set to false?
-b
On 07/15/2014 12:10 PM, Adam Kawa wrote:
> For example, the remote location is configured
> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
> This is why, you see:
> /Logs not available at
> */tmp/logs*/hadoop/logs/application_1405396841766_0003./
>
> PS.
> The full path is configured via
> ${yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>
>
>
> 2014-07-15 18:08 GMT+02:00 Adam Kawa <kawa.adam@gmail.com
> <ma...@gmail.com>>:
>
> IMHO,
> $ yarn logs looks for aggregated logs at remote location.
>
>
>
> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>:
>
> All,
>
> I am running a small cluster with hadoop-2.2.0 installed on an
> NFS shared directory. Since all nodes can access, I do not
> want to enable log aggregation.
>
> My understanding was that if aggregation wasn't enabled, the
> 'yarn logs' command would just look in the
> $HADOOP_HOME/logs/userlogs dir, but that isn't happening:
> [hadoop@host1 ~]$ yarn logs -applicationId
> application_1405396841766_0003
> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to
> ResourceManager at host1/172.17.1.1:8032 <http://172.17.1.1:8032>
> Logs not available at
> /tmp/logs/hadoop/logs/application_1405396841766_0003
> Log aggregation has not completed or is not enabled.
>
> Can anyone suggest what I might be doing wrong?
>
> Thanks,
> Brian
>
>
>
Re: YARN Logs
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Looking at the docs, the yarn.nodemanager.remote-app-log-dir variable
description says "Where to aggregate logs to." But I don't want to use
log aggregation.
Does the 'yarn logs' command look anywhere else if
yarn.log-aggregation-enable is set to false?
-b
On 07/15/2014 12:10 PM, Adam Kawa wrote:
> For example, the remote location is configured
> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
> This is why, you see:
> /Logs not available at
> */tmp/logs*/hadoop/logs/application_1405396841766_0003./
>
> PS.
> The full path is configured via
> ${yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>
>
>
> 2014-07-15 18:08 GMT+02:00 Adam Kawa <kawa.adam@gmail.com
> <ma...@gmail.com>>:
>
> IMHO,
> $ yarn logs looks for aggregated logs at remote location.
>
>
>
> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>:
>
> All,
>
> I am running a small cluster with hadoop-2.2.0 installed on an
> NFS shared directory. Since all nodes can access, I do not
> want to enable log aggregation.
>
> My understanding was that if aggregation wasn't enabled, the
> 'yarn logs' command would just look in the
> $HADOOP_HOME/logs/userlogs dir, but that isn't happening:
> [hadoop@host1 ~]$ yarn logs -applicationId
> application_1405396841766_0003
> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to
> ResourceManager at host1/172.17.1.1:8032 <http://172.17.1.1:8032>
> Logs not available at
> /tmp/logs/hadoop/logs/application_1405396841766_0003
> Log aggregation has not completed or is not enabled.
>
> Can anyone suggest what I might be doing wrong?
>
> Thanks,
> Brian
>
>
>
Re: YARN Logs
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Looking at the docs, the yarn.nodemanager.remote-app-log-dir variable
description says "Where to aggregate logs to." But I don't want to use
log aggregation.
Does the 'yarn logs' command look anywhere else if
yarn.log-aggregation-enable is set to false?
-b
On 07/15/2014 12:10 PM, Adam Kawa wrote:
> For example, the remote location is configured
> via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
> This is why, you see:
> /Logs not available at
> */tmp/logs*/hadoop/logs/application_1405396841766_0003./
>
> PS.
> The full path is configured via
> ${yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
>
>
>
> 2014-07-15 18:08 GMT+02:00 Adam Kawa <kawa.adam@gmail.com
> <ma...@gmail.com>>:
>
> IMHO,
> $ yarn logs looks for aggregated logs at remote location.
>
>
>
> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>:
>
> All,
>
> I am running a small cluster with hadoop-2.2.0 installed on an
> NFS shared directory. Since all nodes can access, I do not
> want to enable log aggregation.
>
> My understanding was that if aggregation wasn't enabled, the
> 'yarn logs' command would just look in the
> $HADOOP_HOME/logs/userlogs dir, but that isn't happening:
> [hadoop@host1 ~]$ yarn logs -applicationId
> application_1405396841766_0003
> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to
> ResourceManager at host1/172.17.1.1:8032 <http://172.17.1.1:8032>
> Logs not available at
> /tmp/logs/hadoop/logs/application_1405396841766_0003
> Log aggregation has not completed or is not enabled.
>
> Can anyone suggest what I might be doing wrong?
>
> Thanks,
> Brian
>
>
>
Re: YARN Logs
Posted by Adam Kawa <ka...@gmail.com>.
For example, the remote location is configured
via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
This is why, you see:
*Logs not available at
/tmp/logs/hadoop/logs/application_1405396841766_0003.*
PS.
The full path is configured via $
{yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
2014-07-15 18:08 GMT+02:00 Adam Kawa <ka...@gmail.com>:
> IMHO,
> $ yarn logs looks for aggregated logs at remote location.
>
>
>
> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bh...@etinternational.com>
> :
>
> All,
>>
>> I am running a small cluster with hadoop-2.2.0 installed on an NFS shared
>> directory. Since all nodes can access, I do not want to enable log
>> aggregation.
>>
>> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
>> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
>> isn't happening:
>> [hadoop@host1 ~]$ yarn logs -applicationId application_1405396841766_0003
>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
>> host1/172.17.1.1:8032
>> Logs not available at /tmp/logs/hadoop/logs/
>> application_1405396841766_0003
>> Log aggregation has not completed or is not enabled.
>>
>> Can anyone suggest what I might be doing wrong?
>>
>> Thanks,
>> Brian
>>
>>
>
Re: YARN Logs
Posted by Adam Kawa <ka...@gmail.com>.
For example, the remote location is configured
via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
This is why, you see:
*Logs not available at
/tmp/logs/hadoop/logs/application_1405396841766_0003.*
PS.
The full path is configured via $
{yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
2014-07-15 18:08 GMT+02:00 Adam Kawa <ka...@gmail.com>:
> IMHO,
> $ yarn logs looks for aggregated logs at remote location.
>
>
>
> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bh...@etinternational.com>
> :
>
> All,
>>
>> I am running a small cluster with hadoop-2.2.0 installed on an NFS shared
>> directory. Since all nodes can access, I do not want to enable log
>> aggregation.
>>
>> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
>> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
>> isn't happening:
>> [hadoop@host1 ~]$ yarn logs -applicationId application_1405396841766_0003
>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
>> host1/172.17.1.1:8032
>> Logs not available at /tmp/logs/hadoop/logs/
>> application_1405396841766_0003
>> Log aggregation has not completed or is not enabled.
>>
>> Can anyone suggest what I might be doing wrong?
>>
>> Thanks,
>> Brian
>>
>>
>
Re: YARN Logs
Posted by Adam Kawa <ka...@gmail.com>.
For example, the remote location is configured
via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
This is why, you see:
*Logs not available at
/tmp/logs/hadoop/logs/application_1405396841766_0003.*
PS.
The full path is configured via $
{yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
2014-07-15 18:08 GMT+02:00 Adam Kawa <ka...@gmail.com>:
> IMHO,
> $ yarn logs looks for aggregated logs at remote location.
>
>
>
> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bh...@etinternational.com>
> :
>
> All,
>>
>> I am running a small cluster with hadoop-2.2.0 installed on an NFS shared
>> directory. Since all nodes can access, I do not want to enable log
>> aggregation.
>>
>> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
>> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
>> isn't happening:
>> [hadoop@host1 ~]$ yarn logs -applicationId application_1405396841766_0003
>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
>> host1/172.17.1.1:8032
>> Logs not available at /tmp/logs/hadoop/logs/
>> application_1405396841766_0003
>> Log aggregation has not completed or is not enabled.
>>
>> Can anyone suggest what I might be doing wrong?
>>
>> Thanks,
>> Brian
>>
>>
>
Re: YARN Logs
Posted by Adam Kawa <ka...@gmail.com>.
For example, the remote location is configured
via yarn.nodemanager.remote-app-log-dir and defaults to "/tmp/logs".
This is why, you see:
*Logs not available at
/tmp/logs/hadoop/logs/application_1405396841766_0003.*
PS.
The full path is configured via $
{yarn.nodemanager.remote-app-log-dir}/${username}/${yarn.nodemanager.remote-app-log-dir-suffix}/${application-id}
2014-07-15 18:08 GMT+02:00 Adam Kawa <ka...@gmail.com>:
> IMHO,
> $ yarn logs looks for aggregated logs at remote location.
>
>
>
> 2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bh...@etinternational.com>
> :
>
> All,
>>
>> I am running a small cluster with hadoop-2.2.0 installed on an NFS shared
>> directory. Since all nodes can access, I do not want to enable log
>> aggregation.
>>
>> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
>> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
>> isn't happening:
>> [hadoop@host1 ~]$ yarn logs -applicationId application_1405396841766_0003
>> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
>> host1/172.17.1.1:8032
>> Logs not available at /tmp/logs/hadoop/logs/
>> application_1405396841766_0003
>> Log aggregation has not completed or is not enabled.
>>
>> Can anyone suggest what I might be doing wrong?
>>
>> Thanks,
>> Brian
>>
>>
>
Re: YARN Logs
Posted by Adam Kawa <ka...@gmail.com>.
IMHO,
$ yarn logs looks for aggregated logs at remote location.
2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bh...@etinternational.com>:
> All,
>
> I am running a small cluster with hadoop-2.2.0 installed on an NFS shared
> directory. Since all nodes can access, I do not want to enable log
> aggregation.
>
> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
> isn't happening:
> [hadoop@host1 ~]$ yarn logs -applicationId application_1405396841766_0003
> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
> host1/172.17.1.1:8032
> Logs not available at /tmp/logs/hadoop/logs/application_1405396841766_0003
> Log aggregation has not completed or is not enabled.
>
> Can anyone suggest what I might be doing wrong?
>
> Thanks,
> Brian
>
>
Re: YARN Logs
Posted by Adam Kawa <ka...@gmail.com>.
IMHO,
$ yarn logs looks for aggregated logs at remote location.
2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bh...@etinternational.com>:
> All,
>
> I am running a small cluster with hadoop-2.2.0 installed on an NFS shared
> directory. Since all nodes can access, I do not want to enable log
> aggregation.
>
> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
> isn't happening:
> [hadoop@host1 ~]$ yarn logs -applicationId application_1405396841766_0003
> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
> host1/172.17.1.1:8032
> Logs not available at /tmp/logs/hadoop/logs/application_1405396841766_0003
> Log aggregation has not completed or is not enabled.
>
> Can anyone suggest what I might be doing wrong?
>
> Thanks,
> Brian
>
>
Re: YARN Logs
Posted by Adam Kawa <ka...@gmail.com>.
IMHO,
$ yarn logs looks for aggregated logs at remote location.
2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bh...@etinternational.com>:
> All,
>
> I am running a small cluster with hadoop-2.2.0 installed on an NFS shared
> directory. Since all nodes can access, I do not want to enable log
> aggregation.
>
> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
> isn't happening:
> [hadoop@host1 ~]$ yarn logs -applicationId application_1405396841766_0003
> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
> host1/172.17.1.1:8032
> Logs not available at /tmp/logs/hadoop/logs/application_1405396841766_0003
> Log aggregation has not completed or is not enabled.
>
> Can anyone suggest what I might be doing wrong?
>
> Thanks,
> Brian
>
>
Re: YARN Logs
Posted by Adam Kawa <ka...@gmail.com>.
IMHO,
$ yarn logs looks for aggregated logs at remote location.
2014-07-15 16:49 GMT+02:00 Brian C. Huffman <bh...@etinternational.com>:
> All,
>
> I am running a small cluster with hadoop-2.2.0 installed on an NFS shared
> directory. Since all nodes can access, I do not want to enable log
> aggregation.
>
> My understanding was that if aggregation wasn't enabled, the 'yarn logs'
> command would just look in the $HADOOP_HOME/logs/userlogs dir, but that
> isn't happening:
> [hadoop@host1 ~]$ yarn logs -applicationId application_1405396841766_0003
> 14/07/15 10:35:58 INFO client.RMProxy: Connecting to ResourceManager at
> host1/172.17.1.1:8032
> Logs not available at /tmp/logs/hadoop/logs/application_1405396841766_0003
> Log aggregation has not completed or is not enabled.
>
> Can anyone suggest what I might be doing wrong?
>
> Thanks,
> Brian
>
>