You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by EdwardKing <zh...@neusoft.com> on 2014/02/17 03:03:26 UTC
Aggregation service start
hadoop 2.2.0, I want to view Tracking UI,so I visit http://172.11.12.6:8088/cluster,
then I click History of Completed Job,such as follows:
MapReduce Job job_1392601388579_0001
Attempt Number Start Time Node Logs
1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
Then I click logs,but it failed.
Aggregation is not enabled. Try the nodemanager at master:8994
I guess it must a service don't start, which command I need to execute under home/software/hadoop-2.2.0/sbin ? Thanks.
[hadoop@node1 sbin]$ ls
distribute-exclude.sh start-all.cmd stop-all.sh
hadoop-daemon.sh start-all.sh stop-balancer.sh
hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
hdfs-config.cmd start-dfs.cmd stop-dfs.sh
hdfs-config.sh start-dfs.sh stop-secure-dns.sh
httpfs.sh start-secure-dns.sh stop-yarn.cmd
mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
slaves.sh stop-all.cmd yarn-daemons.sh
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Aggregation service start
Posted by Zhijie Shen <zs...@hortonworks.com>.
"But when the job complete, then I click History of Tracking UI
http://172.11.12.6:8088/cluster again, it raise following error:
Firefox can't establish a connection to the server at master:19888."
This is a problem other than log aggregation. After an MapReduce job
completes, the tracking URL is pointing to the MapReduce history server. It
is very likely that the history server hasn't been running on your machine,
such that you didn't get response from it.
- Zhijie
On Sun, Feb 16, 2014 at 10:03 PM, EdwardKing <zh...@neusoft.com> wrote:
> Thanks for you help. I set yarn-site.xml as you told me,like follows:
>
> [hadoop@master hadoop]$ cat yarn-site.xml
> <?xml version="1.0"?>
> <!--
> Licensed under the Apache License, Version 2.0 (the "License");
> you may not use this file except in compliance with the License.
> You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License. See accompanying LICENSE file.
> -->
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
> <property>
> <name>yarn.resourcemanager.resource-tracker.address</name>
> <value>master:8990</value>
> <description>host is the hostname of the resource manager and port is
> the port on which the NodeManagers contact the Resource Manager.
> </description>
> </property>
> <property>
> <name>yarn.resourcemanager.scheduler.address</name>
> <value>master:8991</value>
> <description>host is the hostname of the resourcemanager and port is the
> port on which the Applications in the cluster talk to the Resource Manager.
> </description>
> </property>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> <description>In case you do not want to use the default
> scheduler</description>
> </property>
> <property>
> <name>yarn.resourcemanager.address</name>
> <value>master:8993</value>
> <description>the host is the hostname of the ResourceManager and port is
> the port on which the clients can talk to the Resource Manager
> </description>
> </property>
> <property>
> <name>yarn.nodemanager.local-dirs</name>
> <value>/home/software/tmp/node</value>
> <description>the local directions used by the nodemanager</description>
> </property>
> <property>
> <name>yarn.nodemanager.address</name>
> <value>master:8994</value>
> <description>the nodemanagers bind to this port</description>
> </property>
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>5120</value>
> <description>the amount of memory on the NodeManager in GB</description>
> </property>
> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/home/software/tmp/app-logs</value>
> <description>directory on hdfs where the application logs are moved to
> </description>
> </property>
> <property>
> <name>yarn.nodemanager.log-dirs</name>
> <value>/home/software/tmp/node</value>
> <description>the directories used by Nodemanager as log
> directories</description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run
> </description>
> </property>
>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> </configuration>
>
> Then I submit a job,when this job is running, I click History of Tracking
> UI http://172.11.12.6:8088/cluster
> I can view all log information. It runs ok.
> But when the job complete, then I click History of Tracking UI
> http://172.11.12.6:8088/cluster again, it raise following error:
>
> Firefox can't establish a connection to the server at master:19888.
>
> Do I missing some configuration information in my xml file? How to
> correct? Thanks in advance.
>
>
>
>
>
>
>
>
> ----- Original Message -----
> From: Zhijie Shen
> To: user@hadoop.apache.org
> Sent: Monday, February 17, 2014 11:11 AM
> Subject: Re: Aggregation service start
>
>
> Please set
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
> in yarn-site.xml to enable log aggregation.
> -Zhijie
>
> On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
>
> hadoop 2.2.0, I want to view Tracking UI,so I visit
> http://172.11.12.6:8088/cluster,
> then I click History of Completed Job,such as follows:
>
> MapReduce Job job_1392601388579_0001
> Attempt Number Start Time Node Logs
> 1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
>
> Then I click logs,but it failed.
> Aggregation is not enabled. Try the nodemanager at master:8994
>
> I guess it must a service don't start, which command I need to execute
> under home/software/hadoop-2.2.0/sbin ? Thanks.
> [hadoop@node1 sbin]$ ls
> distribute-exclude.sh start-all.cmd stop-all.sh
> hadoop-daemon.sh start-all.sh stop-balancer.sh
> hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
> hdfs-config.cmd start-dfs.cmd stop-dfs.sh
> hdfs-config.sh start-dfs.sh stop-secure-dns.sh
> httpfs.sh start-secure-dns.sh stop-yarn.cmd
> mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
> refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
> slaves.sh stop-all.cmd yarn-daemons.sh
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Aggregation service start
Posted by Zhijie Shen <zs...@hortonworks.com>.
"But when the job complete, then I click History of Tracking UI
http://172.11.12.6:8088/cluster again, it raise following error:
Firefox can't establish a connection to the server at master:19888."
This is a problem other than log aggregation. After an MapReduce job
completes, the tracking URL is pointing to the MapReduce history server. It
is very likely that the history server hasn't been running on your machine,
such that you didn't get response from it.
- Zhijie
On Sun, Feb 16, 2014 at 10:03 PM, EdwardKing <zh...@neusoft.com> wrote:
> Thanks for you help. I set yarn-site.xml as you told me,like follows:
>
> [hadoop@master hadoop]$ cat yarn-site.xml
> <?xml version="1.0"?>
> <!--
> Licensed under the Apache License, Version 2.0 (the "License");
> you may not use this file except in compliance with the License.
> You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License. See accompanying LICENSE file.
> -->
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
> <property>
> <name>yarn.resourcemanager.resource-tracker.address</name>
> <value>master:8990</value>
> <description>host is the hostname of the resource manager and port is
> the port on which the NodeManagers contact the Resource Manager.
> </description>
> </property>
> <property>
> <name>yarn.resourcemanager.scheduler.address</name>
> <value>master:8991</value>
> <description>host is the hostname of the resourcemanager and port is the
> port on which the Applications in the cluster talk to the Resource Manager.
> </description>
> </property>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> <description>In case you do not want to use the default
> scheduler</description>
> </property>
> <property>
> <name>yarn.resourcemanager.address</name>
> <value>master:8993</value>
> <description>the host is the hostname of the ResourceManager and port is
> the port on which the clients can talk to the Resource Manager
> </description>
> </property>
> <property>
> <name>yarn.nodemanager.local-dirs</name>
> <value>/home/software/tmp/node</value>
> <description>the local directions used by the nodemanager</description>
> </property>
> <property>
> <name>yarn.nodemanager.address</name>
> <value>master:8994</value>
> <description>the nodemanagers bind to this port</description>
> </property>
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>5120</value>
> <description>the amount of memory on the NodeManager in GB</description>
> </property>
> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/home/software/tmp/app-logs</value>
> <description>directory on hdfs where the application logs are moved to
> </description>
> </property>
> <property>
> <name>yarn.nodemanager.log-dirs</name>
> <value>/home/software/tmp/node</value>
> <description>the directories used by Nodemanager as log
> directories</description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run
> </description>
> </property>
>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> </configuration>
>
> Then I submit a job,when this job is running, I click History of Tracking
> UI http://172.11.12.6:8088/cluster
> I can view all log information. It runs ok.
> But when the job complete, then I click History of Tracking UI
> http://172.11.12.6:8088/cluster again, it raise following error:
>
> Firefox can't establish a connection to the server at master:19888.
>
> Do I missing some configuration information in my xml file? How to
> correct? Thanks in advance.
>
>
>
>
>
>
>
>
> ----- Original Message -----
> From: Zhijie Shen
> To: user@hadoop.apache.org
> Sent: Monday, February 17, 2014 11:11 AM
> Subject: Re: Aggregation service start
>
>
> Please set
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
> in yarn-site.xml to enable log aggregation.
> -Zhijie
>
> On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
>
> hadoop 2.2.0, I want to view Tracking UI,so I visit
> http://172.11.12.6:8088/cluster,
> then I click History of Completed Job,such as follows:
>
> MapReduce Job job_1392601388579_0001
> Attempt Number Start Time Node Logs
> 1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
>
> Then I click logs,but it failed.
> Aggregation is not enabled. Try the nodemanager at master:8994
>
> I guess it must a service don't start, which command I need to execute
> under home/software/hadoop-2.2.0/sbin ? Thanks.
> [hadoop@node1 sbin]$ ls
> distribute-exclude.sh start-all.cmd stop-all.sh
> hadoop-daemon.sh start-all.sh stop-balancer.sh
> hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
> hdfs-config.cmd start-dfs.cmd stop-dfs.sh
> hdfs-config.sh start-dfs.sh stop-secure-dns.sh
> httpfs.sh start-secure-dns.sh stop-yarn.cmd
> mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
> refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
> slaves.sh stop-all.cmd yarn-daemons.sh
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Aggregation service start
Posted by Zhijie Shen <zs...@hortonworks.com>.
"But when the job complete, then I click History of Tracking UI
http://172.11.12.6:8088/cluster again, it raise following error:
Firefox can't establish a connection to the server at master:19888."
This is a problem other than log aggregation. After an MapReduce job
completes, the tracking URL is pointing to the MapReduce history server. It
is very likely that the history server hasn't been running on your machine,
such that you didn't get response from it.
- Zhijie
On Sun, Feb 16, 2014 at 10:03 PM, EdwardKing <zh...@neusoft.com> wrote:
> Thanks for you help. I set yarn-site.xml as you told me,like follows:
>
> [hadoop@master hadoop]$ cat yarn-site.xml
> <?xml version="1.0"?>
> <!--
> Licensed under the Apache License, Version 2.0 (the "License");
> you may not use this file except in compliance with the License.
> You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License. See accompanying LICENSE file.
> -->
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
> <property>
> <name>yarn.resourcemanager.resource-tracker.address</name>
> <value>master:8990</value>
> <description>host is the hostname of the resource manager and port is
> the port on which the NodeManagers contact the Resource Manager.
> </description>
> </property>
> <property>
> <name>yarn.resourcemanager.scheduler.address</name>
> <value>master:8991</value>
> <description>host is the hostname of the resourcemanager and port is the
> port on which the Applications in the cluster talk to the Resource Manager.
> </description>
> </property>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> <description>In case you do not want to use the default
> scheduler</description>
> </property>
> <property>
> <name>yarn.resourcemanager.address</name>
> <value>master:8993</value>
> <description>the host is the hostname of the ResourceManager and port is
> the port on which the clients can talk to the Resource Manager
> </description>
> </property>
> <property>
> <name>yarn.nodemanager.local-dirs</name>
> <value>/home/software/tmp/node</value>
> <description>the local directions used by the nodemanager</description>
> </property>
> <property>
> <name>yarn.nodemanager.address</name>
> <value>master:8994</value>
> <description>the nodemanagers bind to this port</description>
> </property>
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>5120</value>
> <description>the amount of memory on the NodeManager in GB</description>
> </property>
> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/home/software/tmp/app-logs</value>
> <description>directory on hdfs where the application logs are moved to
> </description>
> </property>
> <property>
> <name>yarn.nodemanager.log-dirs</name>
> <value>/home/software/tmp/node</value>
> <description>the directories used by Nodemanager as log
> directories</description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run
> </description>
> </property>
>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> </configuration>
>
> Then I submit a job,when this job is running, I click History of Tracking
> UI http://172.11.12.6:8088/cluster
> I can view all log information. It runs ok.
> But when the job complete, then I click History of Tracking UI
> http://172.11.12.6:8088/cluster again, it raise following error:
>
> Firefox can't establish a connection to the server at master:19888.
>
> Do I missing some configuration information in my xml file? How to
> correct? Thanks in advance.
>
>
>
>
>
>
>
>
> ----- Original Message -----
> From: Zhijie Shen
> To: user@hadoop.apache.org
> Sent: Monday, February 17, 2014 11:11 AM
> Subject: Re: Aggregation service start
>
>
> Please set
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
> in yarn-site.xml to enable log aggregation.
> -Zhijie
>
> On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
>
> hadoop 2.2.0, I want to view Tracking UI,so I visit
> http://172.11.12.6:8088/cluster,
> then I click History of Completed Job,such as follows:
>
> MapReduce Job job_1392601388579_0001
> Attempt Number Start Time Node Logs
> 1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
>
> Then I click logs,but it failed.
> Aggregation is not enabled. Try the nodemanager at master:8994
>
> I guess it must a service don't start, which command I need to execute
> under home/software/hadoop-2.2.0/sbin ? Thanks.
> [hadoop@node1 sbin]$ ls
> distribute-exclude.sh start-all.cmd stop-all.sh
> hadoop-daemon.sh start-all.sh stop-balancer.sh
> hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
> hdfs-config.cmd start-dfs.cmd stop-dfs.sh
> hdfs-config.sh start-dfs.sh stop-secure-dns.sh
> httpfs.sh start-secure-dns.sh stop-yarn.cmd
> mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
> refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
> slaves.sh stop-all.cmd yarn-daemons.sh
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Aggregation service start
Posted by Zhijie Shen <zs...@hortonworks.com>.
"But when the job complete, then I click History of Tracking UI
http://172.11.12.6:8088/cluster again, it raise following error:
Firefox can't establish a connection to the server at master:19888."
This is a problem other than log aggregation. After an MapReduce job
completes, the tracking URL is pointing to the MapReduce history server. It
is very likely that the history server hasn't been running on your machine,
such that you didn't get response from it.
- Zhijie
On Sun, Feb 16, 2014 at 10:03 PM, EdwardKing <zh...@neusoft.com> wrote:
> Thanks for you help. I set yarn-site.xml as you told me,like follows:
>
> [hadoop@master hadoop]$ cat yarn-site.xml
> <?xml version="1.0"?>
> <!--
> Licensed under the Apache License, Version 2.0 (the "License");
> you may not use this file except in compliance with the License.
> You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License. See accompanying LICENSE file.
> -->
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
> <property>
> <name>yarn.resourcemanager.resource-tracker.address</name>
> <value>master:8990</value>
> <description>host is the hostname of the resource manager and port is
> the port on which the NodeManagers contact the Resource Manager.
> </description>
> </property>
> <property>
> <name>yarn.resourcemanager.scheduler.address</name>
> <value>master:8991</value>
> <description>host is the hostname of the resourcemanager and port is the
> port on which the Applications in the cluster talk to the Resource Manager.
> </description>
> </property>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
> <description>In case you do not want to use the default
> scheduler</description>
> </property>
> <property>
> <name>yarn.resourcemanager.address</name>
> <value>master:8993</value>
> <description>the host is the hostname of the ResourceManager and port is
> the port on which the clients can talk to the Resource Manager
> </description>
> </property>
> <property>
> <name>yarn.nodemanager.local-dirs</name>
> <value>/home/software/tmp/node</value>
> <description>the local directions used by the nodemanager</description>
> </property>
> <property>
> <name>yarn.nodemanager.address</name>
> <value>master:8994</value>
> <description>the nodemanagers bind to this port</description>
> </property>
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>5120</value>
> <description>the amount of memory on the NodeManager in GB</description>
> </property>
> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/home/software/tmp/app-logs</value>
> <description>directory on hdfs where the application logs are moved to
> </description>
> </property>
> <property>
> <name>yarn.nodemanager.log-dirs</name>
> <value>/home/software/tmp/node</value>
> <description>the directories used by Nodemanager as log
> directories</description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run
> </description>
> </property>
>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> </configuration>
>
> Then I submit a job,when this job is running, I click History of Tracking
> UI http://172.11.12.6:8088/cluster
> I can view all log information. It runs ok.
> But when the job complete, then I click History of Tracking UI
> http://172.11.12.6:8088/cluster again, it raise following error:
>
> Firefox can't establish a connection to the server at master:19888.
>
> Do I missing some configuration information in my xml file? How to
> correct? Thanks in advance.
>
>
>
>
>
>
>
>
> ----- Original Message -----
> From: Zhijie Shen
> To: user@hadoop.apache.org
> Sent: Monday, February 17, 2014 11:11 AM
> Subject: Re: Aggregation service start
>
>
> Please set
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
> in yarn-site.xml to enable log aggregation.
> -Zhijie
>
> On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
>
> hadoop 2.2.0, I want to view Tracking UI,so I visit
> http://172.11.12.6:8088/cluster,
> then I click History of Completed Job,such as follows:
>
> MapReduce Job job_1392601388579_0001
> Attempt Number Start Time Node Logs
> 1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
>
> Then I click logs,but it failed.
> Aggregation is not enabled. Try the nodemanager at master:8994
>
> I guess it must a service don't start, which command I need to execute
> under home/software/hadoop-2.2.0/sbin ? Thanks.
> [hadoop@node1 sbin]$ ls
> distribute-exclude.sh start-all.cmd stop-all.sh
> hadoop-daemon.sh start-all.sh stop-balancer.sh
> hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
> hdfs-config.cmd start-dfs.cmd stop-dfs.sh
> hdfs-config.sh start-dfs.sh stop-secure-dns.sh
> httpfs.sh start-secure-dns.sh stop-yarn.cmd
> mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
> refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
> slaves.sh stop-all.cmd yarn-daemons.sh
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Aggregation service start
Posted by EdwardKing <zh...@neusoft.com>.
Thanks for you help. I set yarn-site.xml as you told me,like follows:
[hadoop@master hadoop]$ cat yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8990</value>
<description>host is the hostname of the resource manager and port is the port on which the NodeManagers contact the Resource Manager.
</description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8991</value>
<description>host is the hostname of the resourcemanager and port is the port on which the Applications in the cluster talk to the Resource Manager.
</description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
<description>In case you do not want to use the default scheduler</description>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8993</value>
<description>the host is the hostname of the ResourceManager and port is the port on which the clients can talk to the Resource Manager </description>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/software/tmp/node</value>
<description>the local directions used by the nodemanager</description>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>master:8994</value>
<description>the nodemanagers bind to this port</description>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>5120</value>
<description>the amount of memory on the NodeManager in GB</description>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/home/software/tmp/app-logs</value>
<description>directory on hdfs where the application logs are moved to </description>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/software/tmp/node</value>
<description>the directories used by Nodemanager as log directories</description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>shuffle service that needs to be set for Map Reduce to run </description>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
</configuration>
Then I submit a job,when this job is running, I click History of Tracking UI http://172.11.12.6:8088/cluster
I can view all log information. It runs ok.
But when the job complete, then I click History of Tracking UI http://172.11.12.6:8088/cluster again, it raise following error:
Firefox can't establish a connection to the server at master:19888.
Do I missing some configuration information in my xml file? How to correct? Thanks in advance.
----- Original Message -----
From: Zhijie Shen
To: user@hadoop.apache.org
Sent: Monday, February 17, 2014 11:11 AM
Subject: Re: Aggregation service start
Please set
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
in yarn-site.xml to enable log aggregation.
-Zhijie
On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
hadoop 2.2.0, I want to view Tracking UI,so I visit http://172.11.12.6:8088/cluster,
then I click History of Completed Job,such as follows:
MapReduce Job job_1392601388579_0001
Attempt Number Start Time Node Logs
1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
Then I click logs,but it failed.
Aggregation is not enabled. Try the nodemanager at master:8994
I guess it must a service don't start, which command I need to execute under home/software/hadoop-2.2.0/sbin ? Thanks.
[hadoop@node1 sbin]$ ls
distribute-exclude.sh start-all.cmd stop-all.sh
hadoop-daemon.sh start-all.sh stop-balancer.sh
hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
hdfs-config.cmd start-dfs.cmd stop-dfs.sh
hdfs-config.sh start-dfs.sh stop-secure-dns.sh
httpfs.sh start-secure-dns.sh stop-yarn.cmd
mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
slaves.sh stop-all.cmd yarn-daemons.sh
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Aggregation service start
Posted by EdwardKing <zh...@neusoft.com>.
Thanks for you help. I set yarn-site.xml as you told me,like follows:
[hadoop@master hadoop]$ cat yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8990</value>
<description>host is the hostname of the resource manager and port is the port on which the NodeManagers contact the Resource Manager.
</description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8991</value>
<description>host is the hostname of the resourcemanager and port is the port on which the Applications in the cluster talk to the Resource Manager.
</description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
<description>In case you do not want to use the default scheduler</description>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8993</value>
<description>the host is the hostname of the ResourceManager and port is the port on which the clients can talk to the Resource Manager </description>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/software/tmp/node</value>
<description>the local directions used by the nodemanager</description>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>master:8994</value>
<description>the nodemanagers bind to this port</description>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>5120</value>
<description>the amount of memory on the NodeManager in GB</description>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/home/software/tmp/app-logs</value>
<description>directory on hdfs where the application logs are moved to </description>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/software/tmp/node</value>
<description>the directories used by Nodemanager as log directories</description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>shuffle service that needs to be set for Map Reduce to run </description>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
</configuration>
Then I submit a job,when this job is running, I click History of Tracking UI http://172.11.12.6:8088/cluster
I can view all log information. It runs ok.
But when the job complete, then I click History of Tracking UI http://172.11.12.6:8088/cluster again, it raise following error:
Firefox can't establish a connection to the server at master:19888.
Do I missing some configuration information in my xml file? How to correct? Thanks in advance.
----- Original Message -----
From: Zhijie Shen
To: user@hadoop.apache.org
Sent: Monday, February 17, 2014 11:11 AM
Subject: Re: Aggregation service start
Please set
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
in yarn-site.xml to enable log aggregation.
-Zhijie
On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
hadoop 2.2.0, I want to view Tracking UI,so I visit http://172.11.12.6:8088/cluster,
then I click History of Completed Job,such as follows:
MapReduce Job job_1392601388579_0001
Attempt Number Start Time Node Logs
1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
Then I click logs,but it failed.
Aggregation is not enabled. Try the nodemanager at master:8994
I guess it must a service don't start, which command I need to execute under home/software/hadoop-2.2.0/sbin ? Thanks.
[hadoop@node1 sbin]$ ls
distribute-exclude.sh start-all.cmd stop-all.sh
hadoop-daemon.sh start-all.sh stop-balancer.sh
hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
hdfs-config.cmd start-dfs.cmd stop-dfs.sh
hdfs-config.sh start-dfs.sh stop-secure-dns.sh
httpfs.sh start-secure-dns.sh stop-yarn.cmd
mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
slaves.sh stop-all.cmd yarn-daemons.sh
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Aggregation service start
Posted by EdwardKing <zh...@neusoft.com>.
Thanks for you help. I set yarn-site.xml as you told me,like follows:
[hadoop@master hadoop]$ cat yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8990</value>
<description>host is the hostname of the resource manager and port is the port on which the NodeManagers contact the Resource Manager.
</description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8991</value>
<description>host is the hostname of the resourcemanager and port is the port on which the Applications in the cluster talk to the Resource Manager.
</description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
<description>In case you do not want to use the default scheduler</description>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8993</value>
<description>the host is the hostname of the ResourceManager and port is the port on which the clients can talk to the Resource Manager </description>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/software/tmp/node</value>
<description>the local directions used by the nodemanager</description>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>master:8994</value>
<description>the nodemanagers bind to this port</description>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>5120</value>
<description>the amount of memory on the NodeManager in GB</description>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/home/software/tmp/app-logs</value>
<description>directory on hdfs where the application logs are moved to </description>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/software/tmp/node</value>
<description>the directories used by Nodemanager as log directories</description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>shuffle service that needs to be set for Map Reduce to run </description>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
</configuration>
Then I submit a job,when this job is running, I click History of Tracking UI http://172.11.12.6:8088/cluster
I can view all log information. It runs ok.
But when the job complete, then I click History of Tracking UI http://172.11.12.6:8088/cluster again, it raise following error:
Firefox can't establish a connection to the server at master:19888.
Do I missing some configuration information in my xml file? How to correct? Thanks in advance.
----- Original Message -----
From: Zhijie Shen
To: user@hadoop.apache.org
Sent: Monday, February 17, 2014 11:11 AM
Subject: Re: Aggregation service start
Please set
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
in yarn-site.xml to enable log aggregation.
-Zhijie
On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
hadoop 2.2.0, I want to view Tracking UI,so I visit http://172.11.12.6:8088/cluster,
then I click History of Completed Job,such as follows:
MapReduce Job job_1392601388579_0001
Attempt Number Start Time Node Logs
1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
Then I click logs,but it failed.
Aggregation is not enabled. Try the nodemanager at master:8994
I guess it must a service don't start, which command I need to execute under home/software/hadoop-2.2.0/sbin ? Thanks.
[hadoop@node1 sbin]$ ls
distribute-exclude.sh start-all.cmd stop-all.sh
hadoop-daemon.sh start-all.sh stop-balancer.sh
hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
hdfs-config.cmd start-dfs.cmd stop-dfs.sh
hdfs-config.sh start-dfs.sh stop-secure-dns.sh
httpfs.sh start-secure-dns.sh stop-yarn.cmd
mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
slaves.sh stop-all.cmd yarn-daemons.sh
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Aggregation service start
Posted by EdwardKing <zh...@neusoft.com>.
Thanks for you help. I set yarn-site.xml as you told me,like follows:
[hadoop@master hadoop]$ cat yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8990</value>
<description>host is the hostname of the resource manager and port is the port on which the NodeManagers contact the Resource Manager.
</description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8991</value>
<description>host is the hostname of the resourcemanager and port is the port on which the Applications in the cluster talk to the Resource Manager.
</description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
<description>In case you do not want to use the default scheduler</description>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8993</value>
<description>the host is the hostname of the ResourceManager and port is the port on which the clients can talk to the Resource Manager </description>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/software/tmp/node</value>
<description>the local directions used by the nodemanager</description>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>master:8994</value>
<description>the nodemanagers bind to this port</description>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>5120</value>
<description>the amount of memory on the NodeManager in GB</description>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/home/software/tmp/app-logs</value>
<description>directory on hdfs where the application logs are moved to </description>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/software/tmp/node</value>
<description>the directories used by Nodemanager as log directories</description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>shuffle service that needs to be set for Map Reduce to run </description>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
</configuration>
Then I submit a job,when this job is running, I click History of Tracking UI http://172.11.12.6:8088/cluster
I can view all log information. It runs ok.
But when the job complete, then I click History of Tracking UI http://172.11.12.6:8088/cluster again, it raise following error:
Firefox can't establish a connection to the server at master:19888.
Do I missing some configuration information in my xml file? How to correct? Thanks in advance.
----- Original Message -----
From: Zhijie Shen
To: user@hadoop.apache.org
Sent: Monday, February 17, 2014 11:11 AM
Subject: Re: Aggregation service start
Please set
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
in yarn-site.xml to enable log aggregation.
-Zhijie
On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
hadoop 2.2.0, I want to view Tracking UI,so I visit http://172.11.12.6:8088/cluster,
then I click History of Completed Job,such as follows:
MapReduce Job job_1392601388579_0001
Attempt Number Start Time Node Logs
1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
Then I click logs,but it failed.
Aggregation is not enabled. Try the nodemanager at master:8994
I guess it must a service don't start, which command I need to execute under home/software/hadoop-2.2.0/sbin ? Thanks.
[hadoop@node1 sbin]$ ls
distribute-exclude.sh start-all.cmd stop-all.sh
hadoop-daemon.sh start-all.sh stop-balancer.sh
hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
hdfs-config.cmd start-dfs.cmd stop-dfs.sh
hdfs-config.sh start-dfs.sh stop-secure-dns.sh
httpfs.sh start-secure-dns.sh stop-yarn.cmd
mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
slaves.sh stop-all.cmd yarn-daemons.sh
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Aggregation service start
Posted by Zhijie Shen <zs...@hortonworks.com>.
Please set
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
in yarn-site.xml to enable log aggregation.
-Zhijie
On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
> hadoop 2.2.0, I want to view Tracking UI,so I visit
> http://172.11.12.6:8088/cluster,
> then I click History of Completed Job,such as follows:
>
> MapReduce Job job_1392601388579_0001
> Attempt Number Start Time Node Logs
> 1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
>
> Then I click logs,but it failed.
> Aggregation is not enabled. Try the nodemanager at master:8994
>
> I guess it must a service don't start, which command I need to execute
> under home/software/hadoop-2.2.0/sbin ? Thanks.
> [hadoop@node1 sbin]$ ls
> distribute-exclude.sh start-all.cmd stop-all.sh
> hadoop-daemon.sh start-all.sh stop-balancer.sh
> hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
> hdfs-config.cmd start-dfs.cmd stop-dfs.sh
> hdfs-config.sh start-dfs.sh stop-secure-dns.sh
> httpfs.sh start-secure-dns.sh stop-yarn.cmd
> mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
> refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
> slaves.sh stop-all.cmd yarn-daemons.sh
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Aggregation service start
Posted by Zhijie Shen <zs...@hortonworks.com>.
Please set
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
in yarn-site.xml to enable log aggregation.
-Zhijie
On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
> hadoop 2.2.0, I want to view Tracking UI,so I visit
> http://172.11.12.6:8088/cluster,
> then I click History of Completed Job,such as follows:
>
> MapReduce Job job_1392601388579_0001
> Attempt Number Start Time Node Logs
> 1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
>
> Then I click logs,but it failed.
> Aggregation is not enabled. Try the nodemanager at master:8994
>
> I guess it must a service don't start, which command I need to execute
> under home/software/hadoop-2.2.0/sbin ? Thanks.
> [hadoop@node1 sbin]$ ls
> distribute-exclude.sh start-all.cmd stop-all.sh
> hadoop-daemon.sh start-all.sh stop-balancer.sh
> hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
> hdfs-config.cmd start-dfs.cmd stop-dfs.sh
> hdfs-config.sh start-dfs.sh stop-secure-dns.sh
> httpfs.sh start-secure-dns.sh stop-yarn.cmd
> mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
> refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
> slaves.sh stop-all.cmd yarn-daemons.sh
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Aggregation service start
Posted by Zhijie Shen <zs...@hortonworks.com>.
Please set
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
in yarn-site.xml to enable log aggregation.
-Zhijie
On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
> hadoop 2.2.0, I want to view Tracking UI,so I visit
> http://172.11.12.6:8088/cluster,
> then I click History of Completed Job,such as follows:
>
> MapReduce Job job_1392601388579_0001
> Attempt Number Start Time Node Logs
> 1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
>
> Then I click logs,but it failed.
> Aggregation is not enabled. Try the nodemanager at master:8994
>
> I guess it must a service don't start, which command I need to execute
> under home/software/hadoop-2.2.0/sbin ? Thanks.
> [hadoop@node1 sbin]$ ls
> distribute-exclude.sh start-all.cmd stop-all.sh
> hadoop-daemon.sh start-all.sh stop-balancer.sh
> hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
> hdfs-config.cmd start-dfs.cmd stop-dfs.sh
> hdfs-config.sh start-dfs.sh stop-secure-dns.sh
> httpfs.sh start-secure-dns.sh stop-yarn.cmd
> mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
> refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
> slaves.sh stop-all.cmd yarn-daemons.sh
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Aggregation service start
Posted by Zhijie Shen <zs...@hortonworks.com>.
Please set
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
in yarn-site.xml to enable log aggregation.
-Zhijie
On Feb 16, 2014 6:15 PM, "EdwardKing" <zh...@neusoft.com> wrote:
> hadoop 2.2.0, I want to view Tracking UI,so I visit
> http://172.11.12.6:8088/cluster,
> then I click History of Completed Job,such as follows:
>
> MapReduce Job job_1392601388579_0001
> Attempt Number Start Time Node Logs
> 1 Sun Feb 16 17:44:57 PST 2014 master:8042 logs
>
> Then I click logs,but it failed.
> Aggregation is not enabled. Try the nodemanager at master:8994
>
> I guess it must a service don't start, which command I need to execute
> under home/software/hadoop-2.2.0/sbin ? Thanks.
> [hadoop@node1 sbin]$ ls
> distribute-exclude.sh start-all.cmd stop-all.sh
> hadoop-daemon.sh start-all.sh stop-balancer.sh
> hadoop-daemons.sh start-balancer.sh stop-dfs.cmd
> hdfs-config.cmd start-dfs.cmd stop-dfs.sh
> hdfs-config.sh start-dfs.sh stop-secure-dns.sh
> httpfs.sh start-secure-dns.sh stop-yarn.cmd
> mr-jobhistory-daemon.sh start-yarn.cmd stop-yarn.sh
> refresh-namenodes.sh start-yarn.sh yarn-daemon.sh
> slaves.sh stop-all.cmd yarn-daemons.sh
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.