RE: HBASE Region server failing to start after Kerberos is enabled

["Gangavarapu, Venkata" <Ve...@bcbsa.com>]

Hi,

This got solved.
Problem is with hbase.bulkload.staging.dir directory permissions.
It was not with hbase:hdfs user permissions.
Once it is changed to hbase:hdfs, region servers came up.

Thanks for giving me the hint

-Venkat

From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, June 26, 2015 1:10 PM
To: common-user@hadoop.apache.org
Subject: Re: HBASE Region server failing to start after Kerberos is enabled

Can you post the complete stack trace for 'Failed to get FileSystem instance' ?

What's the permission for /apps/hbase/staging ?

Looking at commit log of SecureBulkLoadEndpoint.java, there have been a lot bug fixes since 0.98.4
Please consider upgrading hbase

Cheers

On Fri, Jun 26, 2015 at 10:48 AM, Gangavarupu, Venkata - Contingent Worker <ve...@bcbsa.com>> wrote:
HI All,

The region servers failing to start, after Kerberos is enabled, with below error.
Hadoop -2.6.0
HBase-0.98.4

2015-06-24 15:58:48,884 DEBUG [RS_OPEN_META-mdcthdpdas06lp:60020-0] regionserver.HRegion: Registered coprocessor service: region=hbase:meta,,1 service=AuthenticationService
2015-06-24 15:58:48,886 INFO  [RS_OPEN_META-mdcthdpdas06lp:60020-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.token.TokenProvider was loaded successfully with priority (536870911).
2015-06-24 15:58:48,894 DEBUG [RS_OPEN_META-mdcthdpdas06lp:60020-0] regionserver.HRegion: Registered coprocessor service: region=hbase:meta,,1 service=SecureBulkLoadService
2015-06-24 15:58:48,907 ERROR [RS_OPEN_META-mdcthdpdas06lp:60020-0] coprocessor.CoprocessorHost: The coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw an unexpected exception
java.lang.IllegalStateException: Failed to get FileSystem instance

I see below properties are included in hbase-site.xml file

<property>
      <name>hbase.coprocessor.region.classes</name>
      <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,org.apache.hadoop.hbase.security.access.AccessController</value>
    </property>

<property>
      <name>hbase.bulkload.staging.dir</name>
      <value>/apps/hbase/staging</value>
    </property>


I deleted org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint from the hbase.coprocessor.region.classes and tried to start. It worked.
I think SecureBulkLoad is causing the problem.

Please help to come over this issue. I would like to have SecureBulkLoad class.

Thanks,
Venkat