Bug in Ambari 1.5 and multi-homed hbase masters

[Chris Tarnas <cf...@biotiquesystems.com>]

Hello,

We need to add java.net.preferIPv4Stack=true to the HBASE_OPTS to overcome an old bug in Hadoop and IPv6, how can we do that? It appears in /etc/hbase/conf/hbase-env.cmd but when you look at the logs for the master startup, it is missing.

We need to tell the master what network interface to use via the hbase-site.xml option "hbase.master.dns.interface". If you specify a specific interface with that and have IPv6 enabled this causes an java.lang.ArrayIndexOutOfBoundsException during master startup. It happens at org.apache.hadoop.net.DNS.reverseDns(DNS.java:78) when it attempts to parse an IPv6 address as a IPv4. (I submitted a patch to the underlying parsing issue via Cloudera two years ago but it appears to have not made it in.)

The work around is to set java.net.preferIPv4Stack=true in the HBASE_OPTS, so back to my original question, how can I do that?

thank you,
-chris

Re: Bug in Ambari 1.5 and multi-homed hbase masters

[Yusaku Sako <yu...@hortonworks.com>]

Currently, Ambari does not expose this HBASE_OPTS parameter via the UI.

I'm assuming you mean hive-env.sh and not hive-env.cmd (.cmd is for
Windows and Ambari does not currently support Windows).

To apply your workaround, you can go to the Ambari Server host, and
edit /var/lib/ambari-server/resources/stacks/HDP/2.0.6/services/HBASE/package/templates/hbase-env.sh.j2.

This is a template file that Ambari uses to generate the actual
hbase-env.sh that gets deployed on your HBase hosts.
This assumes you are using HDP2.  If HDP1, you should edit
/var/lib/ambari-server/esources/stacks/HDP/1.3.2/services/HBASE/package/templates/hbase-env.sh.j2
instead.

After editing the template file, run "ambari-server restart".
Then restart your HBase Service through the Ambari Web UI.  From now
on, hbase-site.xml will have your tweaks.

It is important to keep in mind that you may need to reapply this
workaround when you upgrade Ambari in the future.

Yusaku

On Fri, May 16, 2014 at 3:02 PM, Chris Tarnas <cf...@biotiquesystems.com> wrote:
> Hello,
>
> We need to add java.net.preferIPv4Stack=true to the HBASE_OPTS to overcome an old bug in Hadoop and IPv6, how can we do that? It appears in /etc/hbase/conf/hbase-env.cmd but when you look at the logs for the master startup, it is missing.
>
> We need to tell the master what network interface to use via the hbase-site.xml option "hbase.master.dns.interface". If you specify a specific interface with that and have IPv6 enabled this causes an java.lang.ArrayIndexOutOfBoundsException during master startup. It happens at org.apache.hadoop.net.DNS.reverseDns(DNS.java:78) when it attempts to parse an IPv6 address as a IPv4. (I submitted a patch to the underlying parsing issue via Cloudera two years ago but it appears to have not made it in.)
>
> The work around is to set java.net.preferIPv4Stack=true in the HBASE_OPTS, so back to my original question, how can I do that?
>
> thank you,
> -chris

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.