You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by viral shah <vi...@gmail.com> on 2010/02/17 17:22:42 UTC

Difficulty connecting Hadoop JMX service

I want to monitor my hadoop cluster services using check_jmx nagios plugin.
I use following env. variables in the hadoop-env.sh file
export HADOOP_OPTS=”-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false”
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS=”-Dcom.sun.management.jmxremote
$HADOOP_NAMENODE_OPTS -Dcom.sun.management.jmxremote.port=8004″
export HADOOP_SECONDARYNAMENODE_OPTS=”-Dcom.sun.management.jmxremote
$HADOOP_SECONDARYNAMENODE_OPTS -Dcom.sun.management.jmxremote.port=8005″
export HADOOP_DATANODE_OPTS=”-Dcom.sun.management.jmxremote
$HADOOP_DATANODE_OPTS -Dcom.sun.management.jmxremote.port=8006″
export HADOOP_BALANCER_OPTS=”-Dcom.sun.management.jmxremote
$HADOOP_BALANCER_OPTS -Dcom.sun.management.jmxremote.port=8007″
export HADOOP_JOBTRACKER_OPTS=”-Dcom.sun.management.jmxremote
$HADOOP_JOBTRACKER_OPTS -Dcom.sun.management.jmxremote.port=8008″
export HADOOP_TASKTRACKER_OPTS=”-Dcom.sun.management.jmxremote.port=8009″

but the problem I am facing is that my hadoop machine is behind firewall and
I can't open multiple ports. The JMX RMI connector opens two ports: one is
for the RMI registry, and it's the port that you usually supply with the
-Dcom.sun.management.jmxremote.port=<port> property. The other port is used
to export JMX RMI connection objects. This second port is usually
dynamically allocated at random. So I am not able to connect using Jconsole
or check_jmx plugin.

I tried using example provided at *
http://blogs.sun.com/jmxetc/entry/connecting_through_firewall_using_jmx*, by
changing env. variable like this "export
HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
-Dexample.rmi.agent.port=3000
-javaagent:/root/install/asl-hadoop-0.20.1/lib/CustomAgent.jar
$HADOOP_NAMENODE_OPTS""
The CustomAgent.jar file I created using above mentioned blog entry.
Then when I start hadoop cluster using bin/start-all.sh I get following
error.

root/install/asl-hadoop-0.20.1/bin/hadoop-daemon.sh: line 96:  8983
Aborted                 nohup nice -n $HADOOP_NICENESS
"$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" > "$log"
2>&1 < /dev/null
Create RMI registry on port 3000
Get the platform's MBean server
Initialize the environment map
Create an RMI connector server
Start the RMI connector server on port 3000
service:jmx:rmi://domU-12-31-38-00-B4-F8:3000/jndi/rmi://domU-12-31-38-00-B4-F8:3000/jmxrmi
Create RMI registry on port 3000
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
localhost: starting datanode, logging to
/root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-datanode-domU-12-31-38-00-B4-F8.out
localhost: starting secondarynamenode, logging to
/root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-secondarynamenode-domU-12-31-38-00-B4-F8.out
starting jobtracker, logging to
/root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-jobtracker-domU-12-31-38-00-B4-F8.out
localhost: starting tasktracker, logging to
/root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-tasktracker-domU-12-31-38-00-B4-F8.out

can some help me, what I am doing wrong.

Thanks,
Viral.

Re: Difficulty connecting Hadoop JMX service

Posted by Edward Capriolo <ed...@gmail.com>.
On Wed, Feb 17, 2010 at 11:22 AM, viral shah <vi...@gmail.com> wrote:
> I want to monitor my hadoop cluster services using check_jmx nagios plugin.
> I use following env. variables in the hadoop-env.sh file
> export HADOOP_OPTS=”-Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false”
> # Command specific options appended to HADOOP_OPTS when specified
> export HADOOP_NAMENODE_OPTS=”-Dcom.sun.management.jmxremote
> $HADOOP_NAMENODE_OPTS -Dcom.sun.management.jmxremote.port=8004″
> export HADOOP_SECONDARYNAMENODE_OPTS=”-Dcom.sun.management.jmxremote
> $HADOOP_SECONDARYNAMENODE_OPTS -Dcom.sun.management.jmxremote.port=8005″
> export HADOOP_DATANODE_OPTS=”-Dcom.sun.management.jmxremote
> $HADOOP_DATANODE_OPTS -Dcom.sun.management.jmxremote.port=8006″
> export HADOOP_BALANCER_OPTS=”-Dcom.sun.management.jmxremote
> $HADOOP_BALANCER_OPTS -Dcom.sun.management.jmxremote.port=8007″
> export HADOOP_JOBTRACKER_OPTS=”-Dcom.sun.management.jmxremote
> $HADOOP_JOBTRACKER_OPTS -Dcom.sun.management.jmxremote.port=8008″
> export HADOOP_TASKTRACKER_OPTS=”-Dcom.sun.management.jmxremote.port=8009″
>
> but the problem I am facing is that my hadoop machine is behind firewall and
> I can't open multiple ports. The JMX RMI connector opens two ports: one is
> for the RMI registry, and it's the port that you usually supply with the
> -Dcom.sun.management.jmxremote.port=<port> property. The other port is used
> to export JMX RMI connection objects. This second port is usually
> dynamically allocated at random. So I am not able to connect using Jconsole
> or check_jmx plugin.
>
> I tried using example provided at *
> http://blogs.sun.com/jmxetc/entry/connecting_through_firewall_using_jmx*, by
> changing env. variable like this "export
> HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> -Dexample.rmi.agent.port=3000
> -javaagent:/root/install/asl-hadoop-0.20.1/lib/CustomAgent.jar
> $HADOOP_NAMENODE_OPTS""
> The CustomAgent.jar file I created using above mentioned blog entry.
> Then when I start hadoop cluster using bin/start-all.sh I get following
> error.
>
> root/install/asl-hadoop-0.20.1/bin/hadoop-daemon.sh: line 96:  8983
> Aborted                 nohup nice -n $HADOOP_NICENESS
> "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" > "$log"
> 2>&1 < /dev/null
> Create RMI registry on port 3000
> Get the platform's MBean server
> Initialize the environment map
> Create an RMI connector server
> Start the RMI connector server on port 3000
> service:jmx:rmi://domU-12-31-38-00-B4-F8:3000/jndi/rmi://domU-12-31-38-00-B4-F8:3000/jmxrmi
> Create RMI registry on port 3000
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> localhost: starting datanode, logging to
> /root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-datanode-domU-12-31-38-00-B4-F8.out
> localhost: starting secondarynamenode, logging to
> /root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-secondarynamenode-domU-12-31-38-00-B4-F8.out
> starting jobtracker, logging to
> /root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-jobtracker-domU-12-31-38-00-B4-F8.out
> localhost: starting tasktracker, logging to
> /root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-tasktracker-domU-12-31-38-00-B4-F8.out
>
> can some help me, what I am doing wrong.
>
> Thanks,
> Viral.
>

Yikes,

That is a rather hairy problem.
One possible work around, if you use SSL you might sidestep the RMI
issues. (Do not quote me on that)
http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html

I always chose the course of doing my monitoring from the same subnet
as the cluster to forgo the NAT issues, please do post your findings
as this would be helpful to all doing jmx monitoring.