You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by xeonmailinglist-gmail <xe...@gmail.com> on 2015/03/13 19:06:56 UTC
Can't find map or reduce logs when a job ends.
Hi,
With this configuration in mapreduce (see [1] and [2]), I can’t see the
map and reduce logs of the job when it ends. When I try to look to the
history, I get this error |Not Found: job_1426267326549_0005|. But if I
list the log dir in the hdfs (see [3]), I have some logs about the job,
but not logs about the map or reduce tasks.
Why I can’t see the map and reduce logs? Am I missing some configuration?
[1] configuration in mapred-site.xml
|<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address </name>
<value>hadoop-coc-1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop-coc-1:19888</value>
</property>
<property>
<name>mapreduce.jobhistory.max-age-ms</name>
<value>1800000</value>
</property>
</configuration>
|
[2] configuration in yarn-site.xml
|xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/app-logs</value>
</property>
|
[3] List logs in hadoop
|xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 4 items
drwxrwx--- - xeon supergroup 0 2015-03-13 13:35 /app-logs/xeon/logs/application_1426267324367_0002
drwxrwx--- - xeon supergroup 0 2015-03-13 13:37 /app-logs/xeon/logs/application_1426267324367_0003
drwxrwx--- - xeon supergroup 0 2015-03-13 13:44 /app-logs/xeon/logs/application_1426267324367_0004
drwxrwx--- - xeon supergroup 0 2015-03-13 13:47 /app-logs/xeon/logs/application_1426267324367_0005
|
--
--
Re: Can't find map or reduce logs when a job ends.
Posted by twinkle sachdeva <tw...@gmail.com>.
Hi,
Please try following:
yarn logs -applicationId application_1426267324367_0005
Thanks,
On Tue, Mar 17, 2015 at 8:23 AM, Ranadip Chatterjee <ra...@gmail.com>
wrote:
> Is the job history server up and running on the right host and port?
> Please check the job history server logs, if so? A common reason is for the
> owner of job history server to not have read permission on the logs or for
> the map reduce process owners to not have write permission in the job
> history log location. The error should show up in the job history server
> logs.
>
> Ranadip
> On 13 Mar 2015 18:07, "xeonmailinglist-gmail" <xe...@gmail.com>
> wrote:
>
>> Hi,
>>
>> With this configuration in mapreduce (see [1] and [2]), I can’t see the
>> map and reduce logs of the job when it ends. When I try to look to the
>> history, I get this error Not Found: job_1426267326549_0005. But if I
>> list the log dir in the hdfs (see [3]), I have some logs about the job, but
>> not logs about the map or reduce tasks.
>>
>> Why I can’t see the map and reduce logs? Am I missing some configuration?
>>
>> [1] configuration in mapred-site.xml
>>
>> <configuration>
>> <property>
>> <name>mapreduce.framework.name</name>
>> <value>yarn</value>
>> </property>
>> <property>
>> <name>mapreduce.jobhistory.address </name>
>> <value>hadoop-coc-1:10020</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.jobhistory.webapp.address</name>
>> <value>hadoop-coc-1:19888</value>
>> </property>
>> <property>
>> <name>mapreduce.jobhistory.max-age-ms</name>
>> <value>1800000</value>
>> </property>
>> </configuration>
>>
>> [2] configuration in yarn-site.xml
>>
>> xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml
>> <?xml version="1.0"?>
>> <configuration>
>>
>> <!-- Site specific YARN configuration properties -->
>> <property>
>> <name>yarn.nodemanager.aux-services</name>
>> <value>mapreduce_shuffle</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>> </property>
>> <property>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.remote-app-log-dir</name>
>> <value>/app-logs</value>
>> </property>
>>
>> [3] List logs in hadoop
>>
>> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/
>> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
>> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
>> 15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>> Found 4 items
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:35 /app-logs/xeon/logs/application_1426267324367_0002
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:37 /app-logs/xeon/logs/application_1426267324367_0003
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:44 /app-logs/xeon/logs/application_1426267324367_0004
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:47 /app-logs/xeon/logs/application_1426267324367_0005
>>
>>
>>
>> --
>> --
>>
>>
Re: Can't find map or reduce logs when a job ends.
Posted by twinkle sachdeva <tw...@gmail.com>.
Hi,
Please try following:
yarn logs -applicationId application_1426267324367_0005
Thanks,
On Tue, Mar 17, 2015 at 8:23 AM, Ranadip Chatterjee <ra...@gmail.com>
wrote:
> Is the job history server up and running on the right host and port?
> Please check the job history server logs, if so? A common reason is for the
> owner of job history server to not have read permission on the logs or for
> the map reduce process owners to not have write permission in the job
> history log location. The error should show up in the job history server
> logs.
>
> Ranadip
> On 13 Mar 2015 18:07, "xeonmailinglist-gmail" <xe...@gmail.com>
> wrote:
>
>> Hi,
>>
>> With this configuration in mapreduce (see [1] and [2]), I can’t see the
>> map and reduce logs of the job when it ends. When I try to look to the
>> history, I get this error Not Found: job_1426267326549_0005. But if I
>> list the log dir in the hdfs (see [3]), I have some logs about the job, but
>> not logs about the map or reduce tasks.
>>
>> Why I can’t see the map and reduce logs? Am I missing some configuration?
>>
>> [1] configuration in mapred-site.xml
>>
>> <configuration>
>> <property>
>> <name>mapreduce.framework.name</name>
>> <value>yarn</value>
>> </property>
>> <property>
>> <name>mapreduce.jobhistory.address </name>
>> <value>hadoop-coc-1:10020</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.jobhistory.webapp.address</name>
>> <value>hadoop-coc-1:19888</value>
>> </property>
>> <property>
>> <name>mapreduce.jobhistory.max-age-ms</name>
>> <value>1800000</value>
>> </property>
>> </configuration>
>>
>> [2] configuration in yarn-site.xml
>>
>> xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml
>> <?xml version="1.0"?>
>> <configuration>
>>
>> <!-- Site specific YARN configuration properties -->
>> <property>
>> <name>yarn.nodemanager.aux-services</name>
>> <value>mapreduce_shuffle</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>> </property>
>> <property>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.remote-app-log-dir</name>
>> <value>/app-logs</value>
>> </property>
>>
>> [3] List logs in hadoop
>>
>> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/
>> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
>> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
>> 15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>> Found 4 items
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:35 /app-logs/xeon/logs/application_1426267324367_0002
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:37 /app-logs/xeon/logs/application_1426267324367_0003
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:44 /app-logs/xeon/logs/application_1426267324367_0004
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:47 /app-logs/xeon/logs/application_1426267324367_0005
>>
>>
>>
>> --
>> --
>>
>>
Re: Can't find map or reduce logs when a job ends.
Posted by twinkle sachdeva <tw...@gmail.com>.
Hi,
Please try following:
yarn logs -applicationId application_1426267324367_0005
Thanks,
On Tue, Mar 17, 2015 at 8:23 AM, Ranadip Chatterjee <ra...@gmail.com>
wrote:
> Is the job history server up and running on the right host and port?
> Please check the job history server logs, if so? A common reason is for the
> owner of job history server to not have read permission on the logs or for
> the map reduce process owners to not have write permission in the job
> history log location. The error should show up in the job history server
> logs.
>
> Ranadip
> On 13 Mar 2015 18:07, "xeonmailinglist-gmail" <xe...@gmail.com>
> wrote:
>
>> Hi,
>>
>> With this configuration in mapreduce (see [1] and [2]), I can’t see the
>> map and reduce logs of the job when it ends. When I try to look to the
>> history, I get this error Not Found: job_1426267326549_0005. But if I
>> list the log dir in the hdfs (see [3]), I have some logs about the job, but
>> not logs about the map or reduce tasks.
>>
>> Why I can’t see the map and reduce logs? Am I missing some configuration?
>>
>> [1] configuration in mapred-site.xml
>>
>> <configuration>
>> <property>
>> <name>mapreduce.framework.name</name>
>> <value>yarn</value>
>> </property>
>> <property>
>> <name>mapreduce.jobhistory.address </name>
>> <value>hadoop-coc-1:10020</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.jobhistory.webapp.address</name>
>> <value>hadoop-coc-1:19888</value>
>> </property>
>> <property>
>> <name>mapreduce.jobhistory.max-age-ms</name>
>> <value>1800000</value>
>> </property>
>> </configuration>
>>
>> [2] configuration in yarn-site.xml
>>
>> xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml
>> <?xml version="1.0"?>
>> <configuration>
>>
>> <!-- Site specific YARN configuration properties -->
>> <property>
>> <name>yarn.nodemanager.aux-services</name>
>> <value>mapreduce_shuffle</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>> </property>
>> <property>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.remote-app-log-dir</name>
>> <value>/app-logs</value>
>> </property>
>>
>> [3] List logs in hadoop
>>
>> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/
>> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
>> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
>> 15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>> Found 4 items
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:35 /app-logs/xeon/logs/application_1426267324367_0002
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:37 /app-logs/xeon/logs/application_1426267324367_0003
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:44 /app-logs/xeon/logs/application_1426267324367_0004
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:47 /app-logs/xeon/logs/application_1426267324367_0005
>>
>>
>>
>> --
>> --
>>
>>
Re: Can't find map or reduce logs when a job ends.
Posted by twinkle sachdeva <tw...@gmail.com>.
Hi,
Please try following:
yarn logs -applicationId application_1426267324367_0005
Thanks,
On Tue, Mar 17, 2015 at 8:23 AM, Ranadip Chatterjee <ra...@gmail.com>
wrote:
> Is the job history server up and running on the right host and port?
> Please check the job history server logs, if so? A common reason is for the
> owner of job history server to not have read permission on the logs or for
> the map reduce process owners to not have write permission in the job
> history log location. The error should show up in the job history server
> logs.
>
> Ranadip
> On 13 Mar 2015 18:07, "xeonmailinglist-gmail" <xe...@gmail.com>
> wrote:
>
>> Hi,
>>
>> With this configuration in mapreduce (see [1] and [2]), I can’t see the
>> map and reduce logs of the job when it ends. When I try to look to the
>> history, I get this error Not Found: job_1426267326549_0005. But if I
>> list the log dir in the hdfs (see [3]), I have some logs about the job, but
>> not logs about the map or reduce tasks.
>>
>> Why I can’t see the map and reduce logs? Am I missing some configuration?
>>
>> [1] configuration in mapred-site.xml
>>
>> <configuration>
>> <property>
>> <name>mapreduce.framework.name</name>
>> <value>yarn</value>
>> </property>
>> <property>
>> <name>mapreduce.jobhistory.address </name>
>> <value>hadoop-coc-1:10020</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.jobhistory.webapp.address</name>
>> <value>hadoop-coc-1:19888</value>
>> </property>
>> <property>
>> <name>mapreduce.jobhistory.max-age-ms</name>
>> <value>1800000</value>
>> </property>
>> </configuration>
>>
>> [2] configuration in yarn-site.xml
>>
>> xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml
>> <?xml version="1.0"?>
>> <configuration>
>>
>> <!-- Site specific YARN configuration properties -->
>> <property>
>> <name>yarn.nodemanager.aux-services</name>
>> <value>mapreduce_shuffle</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>> </property>
>> <property>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.remote-app-log-dir</name>
>> <value>/app-logs</value>
>> </property>
>>
>> [3] List logs in hadoop
>>
>> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/
>> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
>> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
>> 15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>> Found 4 items
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:35 /app-logs/xeon/logs/application_1426267324367_0002
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:37 /app-logs/xeon/logs/application_1426267324367_0003
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:44 /app-logs/xeon/logs/application_1426267324367_0004
>> drwxrwx--- - xeon supergroup 0 2015-03-13 13:47 /app-logs/xeon/logs/application_1426267324367_0005
>>
>>
>>
>> --
>> --
>>
>>
Re: Can't find map or reduce logs when a job ends.
Posted by Ranadip Chatterjee <ra...@gmail.com>.
Is the job history server up and running on the right host and port? Please
check the job history server logs, if so? A common reason is for the owner
of job history server to not have read permission on the logs or for the
map reduce process owners to not have write permission in the job history
log location. The error should show up in the job history server logs.
Ranadip
On 13 Mar 2015 18:07, "xeonmailinglist-gmail" <xe...@gmail.com>
wrote:
> Hi,
>
> With this configuration in mapreduce (see [1] and [2]), I can’t see the
> map and reduce logs of the job when it ends. When I try to look to the
> history, I get this error Not Found: job_1426267326549_0005. But if I
> list the log dir in the hdfs (see [3]), I have some logs about the job, but
> not logs about the map or reduce tasks.
>
> Why I can’t see the map and reduce logs? Am I missing some configuration?
>
> [1] configuration in mapred-site.xml
>
> <configuration>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name>mapreduce.jobhistory.address </name>
> <value>hadoop-coc-1:10020</value>
> </property>
>
> <property>
> <name>mapreduce.jobhistory.webapp.address</name>
> <value>hadoop-coc-1:19888</value>
> </property>
> <property>
> <name>mapreduce.jobhistory.max-age-ms</name>
> <value>1800000</value>
> </property>
> </configuration>
>
> [2] configuration in yarn-site.xml
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml
> <?xml version="1.0"?>
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/app-logs</value>
> </property>
>
> [3] List logs in hadoop
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/
> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
> 15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Found 4 items
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:35 /app-logs/xeon/logs/application_1426267324367_0002
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:37 /app-logs/xeon/logs/application_1426267324367_0003
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:44 /app-logs/xeon/logs/application_1426267324367_0004
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:47 /app-logs/xeon/logs/application_1426267324367_0005
>
>
>
> --
> --
>
>
Re: Can't find map or reduce logs when a job ends.
Posted by Ranadip Chatterjee <ra...@gmail.com>.
Is the job history server up and running on the right host and port? Please
check the job history server logs, if so? A common reason is for the owner
of job history server to not have read permission on the logs or for the
map reduce process owners to not have write permission in the job history
log location. The error should show up in the job history server logs.
Ranadip
On 13 Mar 2015 18:07, "xeonmailinglist-gmail" <xe...@gmail.com>
wrote:
> Hi,
>
> With this configuration in mapreduce (see [1] and [2]), I can’t see the
> map and reduce logs of the job when it ends. When I try to look to the
> history, I get this error Not Found: job_1426267326549_0005. But if I
> list the log dir in the hdfs (see [3]), I have some logs about the job, but
> not logs about the map or reduce tasks.
>
> Why I can’t see the map and reduce logs? Am I missing some configuration?
>
> [1] configuration in mapred-site.xml
>
> <configuration>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name>mapreduce.jobhistory.address </name>
> <value>hadoop-coc-1:10020</value>
> </property>
>
> <property>
> <name>mapreduce.jobhistory.webapp.address</name>
> <value>hadoop-coc-1:19888</value>
> </property>
> <property>
> <name>mapreduce.jobhistory.max-age-ms</name>
> <value>1800000</value>
> </property>
> </configuration>
>
> [2] configuration in yarn-site.xml
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml
> <?xml version="1.0"?>
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/app-logs</value>
> </property>
>
> [3] List logs in hadoop
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/
> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
> 15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Found 4 items
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:35 /app-logs/xeon/logs/application_1426267324367_0002
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:37 /app-logs/xeon/logs/application_1426267324367_0003
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:44 /app-logs/xeon/logs/application_1426267324367_0004
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:47 /app-logs/xeon/logs/application_1426267324367_0005
>
>
>
> --
> --
>
>
Re: Can't find map or reduce logs when a job ends.
Posted by Ranadip Chatterjee <ra...@gmail.com>.
Is the job history server up and running on the right host and port? Please
check the job history server logs, if so? A common reason is for the owner
of job history server to not have read permission on the logs or for the
map reduce process owners to not have write permission in the job history
log location. The error should show up in the job history server logs.
Ranadip
On 13 Mar 2015 18:07, "xeonmailinglist-gmail" <xe...@gmail.com>
wrote:
> Hi,
>
> With this configuration in mapreduce (see [1] and [2]), I can’t see the
> map and reduce logs of the job when it ends. When I try to look to the
> history, I get this error Not Found: job_1426267326549_0005. But if I
> list the log dir in the hdfs (see [3]), I have some logs about the job, but
> not logs about the map or reduce tasks.
>
> Why I can’t see the map and reduce logs? Am I missing some configuration?
>
> [1] configuration in mapred-site.xml
>
> <configuration>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name>mapreduce.jobhistory.address </name>
> <value>hadoop-coc-1:10020</value>
> </property>
>
> <property>
> <name>mapreduce.jobhistory.webapp.address</name>
> <value>hadoop-coc-1:19888</value>
> </property>
> <property>
> <name>mapreduce.jobhistory.max-age-ms</name>
> <value>1800000</value>
> </property>
> </configuration>
>
> [2] configuration in yarn-site.xml
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml
> <?xml version="1.0"?>
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/app-logs</value>
> </property>
>
> [3] List logs in hadoop
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/
> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
> 15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Found 4 items
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:35 /app-logs/xeon/logs/application_1426267324367_0002
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:37 /app-logs/xeon/logs/application_1426267324367_0003
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:44 /app-logs/xeon/logs/application_1426267324367_0004
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:47 /app-logs/xeon/logs/application_1426267324367_0005
>
>
>
> --
> --
>
>
Re: Can't find map or reduce logs when a job ends.
Posted by Ranadip Chatterjee <ra...@gmail.com>.
Is the job history server up and running on the right host and port? Please
check the job history server logs, if so? A common reason is for the owner
of job history server to not have read permission on the logs or for the
map reduce process owners to not have write permission in the job history
log location. The error should show up in the job history server logs.
Ranadip
On 13 Mar 2015 18:07, "xeonmailinglist-gmail" <xe...@gmail.com>
wrote:
> Hi,
>
> With this configuration in mapreduce (see [1] and [2]), I can’t see the
> map and reduce logs of the job when it ends. When I try to look to the
> history, I get this error Not Found: job_1426267326549_0005. But if I
> list the log dir in the hdfs (see [3]), I have some logs about the job, but
> not logs about the map or reduce tasks.
>
> Why I can’t see the map and reduce logs? Am I missing some configuration?
>
> [1] configuration in mapred-site.xml
>
> <configuration>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name>mapreduce.jobhistory.address </name>
> <value>hadoop-coc-1:10020</value>
> </property>
>
> <property>
> <name>mapreduce.jobhistory.webapp.address</name>
> <value>hadoop-coc-1:19888</value>
> </property>
> <property>
> <name>mapreduce.jobhistory.max-age-ms</name>
> <value>1800000</value>
> </property>
> </configuration>
>
> [2] configuration in yarn-site.xml
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml
> <?xml version="1.0"?>
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/app-logs</value>
> </property>
>
> [3] List logs in hadoop
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/
> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
> 15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Found 4 items
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:35 /app-logs/xeon/logs/application_1426267324367_0002
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:37 /app-logs/xeon/logs/application_1426267324367_0003
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:44 /app-logs/xeon/logs/application_1426267324367_0004
> drwxrwx--- - xeon supergroup 0 2015-03-13 13:47 /app-logs/xeon/logs/application_1426267324367_0005
>
>
>
> --
> --
>
>