You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Dmitry Lysnichenko (JIRA)" <ji...@apache.org> on 2014/03/26 17:52:17 UTC
[jira] [Created] (AMBARI-5228) gmond processes for master
components are not starting after upgrade
Dmitry Lysnichenko created AMBARI-5228:
------------------------------------------
Summary: gmond processes for master components are not starting after upgrade
Key: AMBARI-5228
URL: https://issues.apache.org/jira/browse/AMBARI-5228
Project: Ambari
Issue Type: Bug
Components: controller, test
Affects Versions: 1.5.0
Reporter: Dmitry Lysnichenko
Assignee: Dmitry Lysnichenko
Priority: Critical
Fix For: 1.5.0
This is an upgrade from 1.4.3 to 1.5.0 and HDP-1.3.2
After upgrade the gmond processes for master components are not staring. The error, when tried to do it manually:
{noformat}
/usr/sbin/gmond --conf=/etc/ganglia/hdp/HDPHBaseMaster/gmond.core.conf -d1
Unable to create tcp_accept_channel. Exiting.
{noformat}
Looks like, not sure why, the presence of gmond.master.conf file is creating trouble. At /etc/ganglia/hdp/HDPHBaseMaster/conf.d
{noformat}
rw-r--r-- 1 root hadoop2 442 Mar 22 06:18 gmond.master.conf
-rw-r--r-- 1 root hadoop2 671 Mar 22 13:39 gmond.slave.conf
{noformat}
If the files are removed - did it for JT and HBaseMaster then
{noformat}
ps aux | grep gmond
nobody 5868 1.6 0.0 59108 1860 ? Ssl 17:57 0:00 /usr/sbin/gmond --conf=/etc/ganglia/hdp/HDPTaskTracker/gmond.core.conf --pid-file=/var/run/ganglia/hdp/HDPTaskTracker/gmond.pid
nobody 5894 1.5 0.0 59108 1864 ? Ssl 17:57 0:00 /usr/sbin/gmond --conf=/etc/ganglia/hdp/HDPJobTracker/gmond.core.conf --pid-file=/var/run/ganglia/hdp/HDPJobTracker/gmond.pid
nobody 5913 1.6 0.1 125200 2200 ? Ssl 17:57 0:00 /usr/sbin/gmond --conf=/etc/ganglia/hdp/HDPSlaves/gmond.core.conf --pid-file=/var/run/ganglia/hdp/HDPSlaves/gmond.pid
nobody 5932 1.6 0.0 59108 1828 ? Ssl 17:57 0:00 /usr/sbin/gmond --conf=/etc/ganglia/hdp/HDPDataNode/gmond.core.conf --pid-file=/var/run/ganglia/hdp/HDPDataNode/gmond.pid
nobody 5952 1.6 0.0 59108 1840 ? Ssl 17:57 0:00 /usr/sbin/gmond --conf=/etc/ganglia/hdp/HDPHistoryServer/gmond.core.conf --pid-file=/var/run/ganglia/hdp/HDPHistoryServer/gmond.pid
nobody 5971 1.6 0.0 59108 1824 ? Ssl 17:57 0:00 /usr/sbin/gmond --conf=/etc/ganglia/hdp/HDPHBaseRegionServer/gmond.core.conf --pid-file=/var/run/ganglia/hdp/HDPHBaseRegionServer/gmond.pid
{noformat}
Notice that gmond for NameNode is not up as the file was not removed.
h2. Details:
At 1.4.x versions, we generated configs following this principle:
- HDPNameNode/gmond.core.conf contains tcp_accept_channel that binds to localhost ( resolves to 127.0.0.1)
- HDPNameNode/conf.d/gmond.master.conf contains udp_recv_channel and tcp_accept_channel that bind to machine hostname (resolves to real ip)
- HDPNameNode/conf.d/gmond.slave.conf contains udp_send_channel that binds to machine hostname (resolves to real ip)
At 1.5.x versions, we changed that:
- HDPNameNode/gmond.core.conf contains tcp_accept_channel that binds to any interface ( 0.0.0.0 )
- HDPNameNode/conf.d/gmond.master.conf contains udp_recv_channel that binds to machine hostname (resolves to real ip). Note that we now have no any tcp_accept_channel definition or it would conflict with definition at gmond.core.conf
- HDPNameNode/conf.d/gmond.slave.conf contains udp_send_channel that binds to machine hostname (resolves to real ip). No changes here.
When we upgrade from 1.4.1 to 1.5.x, these definitions get mixed.
In fact, gmond configure() is called during install and start, and gmetad configure() (that generates gmond.master.conf) is called only during install.
--
This message was sent by Atlassian JIRA
(v6.2#6252)