You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Omkar Joshi <Om...@lntinfotech.com> on 2013/04/15 05:42:03 UTC

Unable to connect from windows desktop to HBase

Hi,

I'm trying to connect from a Java client running on a Windows desktop machine to a remote HBase cluster running in the distributed mode(atop 2-node Hadoop cluster).

1.      On the master(namenode)node :
hduser@cldx-1139-1033:~/hadoop_ecosystem/apache_hbase/hbase_installation/hbase-0.94.6.1/bin$ jps
9161 HMaster
4536 SecondaryNameNode
4368 DataNode
4645 JobTracker
8395 Jps
4813 TaskTracker
4179 NameNode
7717 Main



2.       On the slave(datanode)node :


hduser@cldx-1140-1034:~$ jps

5677 HRegionServer

5559 HQuorumPeer

2634 TaskTracker

3260 Jps

2496 DataNode



3.       I also connected to the shell and created a hbase table and also able to scan it :


hbase(main):004:0> scan 'CUSTOMERS'

ROW                              COLUMN+CELL

 CUSTID12345                     column=CUSTOMER_INFO:EMAIL, timestamp=1365600369284, value=Omkar.Joshi@lntinfotech.com

 CUSTID12345                     column=CUSTOMER_INFO:NAME, timestamp=1365600052104, value=Omkar Joshi

 CUSTID614                       column=CUSTOMER_INFO:NAME, timestamp=1365601350972, value=Prachi Shah

2 row(s) in 0.8760 seconds



4.       The hbase-site.xml has the following configurations:


<configuration>

<property>

    <name>hbase.zookeeper.quorum</name>

    <value>cldx-1140-1034</value>

    <description>The directory shared by RegionServers.

    </description>

</property>

<property>

    <name>hbase.zookeeper.property.dataDir</name>

    <value>/home/hduser/hadoop_ecosystem/apache_hbase/zk_datadir</value>

    <description>Property from ZooKeeper's config zoo.cfg.

    The directory where the snapshot is stored.

    </description>

</property>

<property>

    <name>hbase.rootdir</name>

    <value>hdfs://cldx-1139-1033:9000/hbase</value>

    <description>The directory shared by RegionServers.</description>

</property>

<property>

    <name>hbase.cluster.distributed</name>

    <value>true</value>

    <description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)</description>

</property>

</configuration>



5.       The Hadoop's core-site.xml has the following settings :


<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://cldx-1139-1033:9000</value>

</property>

</configuration>



6.       My java client class is :


package client.hbase;



import java.io.IOException;

import java.util.List;



import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.hbase.HBaseConfiguration;

import org.apache.hadoop.hbase.HColumnDescriptor;

import org.apache.hadoop.hbase.HTableDescriptor;

import org.apache.hadoop.hbase.client.HBaseAdmin;

import org.apache.hadoop.hbase.client.HTableInterface;

import org.apache.hadoop.hbase.client.HTablePool;

import org.apache.hadoop.hbase.client.Put;

import org.apache.hadoop.hbase.client.Result;

import org.apache.hadoop.hbase.client.ResultScanner;

import org.apache.hadoop.hbase.client.Scan;

import org.apache.hadoop.hbase.filter.CompareFilter;

import org.apache.hadoop.hbase.filter.Filter;

import org.apache.hadoop.hbase.filter.RegexStringComparator;

import org.apache.hadoop.hbase.filter.ValueFilter;

import org.apache.hadoop.hbase.util.Bytes;



public class HBaseCRUD {



    private static Configuration config;

    private static HBaseAdmin hbaseAdmin;



    private static HTablePool hTablePool;



    static {

        config = HBaseConfiguration.create();

        config.set("hbase.zookeeper.quorum", "172.25.6.69");



        hTablePool = new HTablePool(config, 2);

    }



    /**

     * @param args

     * @throws IOException

     */

    public static void main(String[] args) throws IOException {

        // TODO Auto-generated method stub



        HBaseCRUD hbaseCRUD = new HBaseCRUD();



        /*hbaseCRUD.createTables("CUSTOMERS", "CUSTOMER_INFO");



        hbaseCRUD.populateTableData("CUSTOMERS");*/



        hbaseCRUD.scanTable("CUSTOMERS", "CUSTOMER_INFO", "EMAIL");

    }



    private void createTables(String tableName, String... columnFamilyNames)

            throws IOException {

        // TODO Auto-generated method stub



        HTableDescriptor tableDesc = new HTableDescriptor(tableName);



        if (!(columnFamilyNames == null || columnFamilyNames.length == 0)) {



            for (String columnFamilyName : columnFamilyNames) {



                HColumnDescriptor columnFamily = new HColumnDescriptor(

                        columnFamilyName);



                tableDesc.addFamily(columnFamily);

            }

        }



        hbaseAdmin.createTable(tableDesc);

    }



    private void populateTableData(String tableName) throws IOException {

        HTableInterface tbl = hTablePool.getTable(Bytes.toBytes(tableName));



        List<Put> tblRows = getTableData(tableName);



        tbl.close();

    }



    private List<Put> getTableData(String tableName) {

        // TODO Auto-generated method stub



        if (tableName == null || tableName.isEmpty())

            return null;



        /* Pull data from wherever located */



        if (tableName.equalsIgnoreCase("CUSTOMERS")) {



            /*

             * Put p1 = new Put(); p1.add(Bytes.toBytes("CUSTOMER_INFO"),

             * Bytes.toBytes("NAME"), value)

             */

        }



        return null;

    }



    private void scanTable(String tableName, String columnFamilyName,

            String... columnNames) throws IOException {



        System.out.println("In HBaseCRUD.scanTable(...)");



        Scan scan = new Scan();

        if (!(columnNames == null || columnNames.length <= 0)) {



            for (String columnName : columnNames) {



                scan.addColumn(Bytes.toBytes(columnFamilyName),

                        Bytes.toBytes(columnName));

            }



            Filter filter = new ValueFilter(CompareFilter.CompareOp.EQUAL,

                    new RegexStringComparator("lntinfotech"));

            scan.setFilter(filter);

        }



        HTableInterface tbl = hTablePool.getTable(Bytes.toBytes(tableName));

        ResultScanner scanResults = tbl.getScanner(scan);



        for (Result result : scanResults) {

            System.out.println("The result is " + result);

        }



        tbl.close();

    }

}



7.       The exception I get is :


In HBaseCRUD.scanTable(...)

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Apr 10, 2013 4:24:54 PM org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper <init>

INFO: The identifier of this process is 3648@INFVA03351

Apr 10, 2013 4:24:56 PM org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow

WARNING: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

Apr 10, 2013 4:24:56 PM org.apache.hadoop.hbase.util.RetryCounter sleepUntilNextRetry

INFO: Sleeping 2000ms before retry #1...

Apr 10, 2013 4:24:58 PM org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow

WARNING: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

Apr 10, 2013 4:24:58 PM org.apache.hadoop.hbase.util.RetryCounter sleepUntilNextRetry

INFO: Sleeping 4000ms before retry #2...


I have also made the master's entry in my local machine's host file. What can be the error?

Thanks and regards !


________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"

RE: Unable to connect from windows desktop to HBase

Posted by Omkar Joshi <Om...@lntinfotech.com>.
Hi Azuryy,

Thanks for the detailed replies !

Regards,
Omkar Joshi
Technology Office

-----Original Message-----
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: Monday, April 15, 2013 4:11 PM
To: user@hbase.apache.org
Subject: Re: Unable to connect from windows desktop to HBase

you need to put  protobuf-java-2.4.1.jar under the hbase_home/lib, then
don't include it in your client class path. but if you used hbase-0.95 or
hbase-0.96.1, protobuf jar is also needed on your client's classpath.



On Mon, Apr 15, 2013 at 6:37 PM, Azuryy Yu <az...@gmail.com> wrote:

>  config = HBaseConfiguration.create();
>
> this will read hbase-site.xml to get hmaster and zookeeper quorums, if
> not, then zookeeper quorums and hbase.master are all empty. so you cannot
> get connection.
>
>
>
>
> On Mon, Apr 15, 2013 at 6:29 PM, Omkar Joshi <Om...@lntinfotech.com>wrote:
>
>> Hi Azuryy,
>>
>> Thanks for the reply !
>>
>> It solved the issue but I'm not clear as to why this is required; also
>> had to include the Google's protobuf-java-2.4.1.jar whose purpose is
>> unclear to me.
>>
>> Regards,
>> Omkar Joshi
>>
>>
>> -----Original Message-----
>> From: Azuryy Yu [mailto:azuryyyu@gmail.com]
>> Sent: Monday, April 15, 2013 3:22 PM
>> To: user@hbase.apache.org
>> Subject: Re: Unable to connect from windows desktop to HBase
>>
>> you should put hbase-site.xml under your class path in your java client.
>>
>> then, don't need to specify zookeeper quorum in your code.
>>
>>
>> On Mon, Apr 15, 2013 at 5:35 PM, Omkar Joshi <Omkar.Joshi@lntinfotech.com
>> >wrote:
>>
>> > Attaching the dumps from the HBase UI :
>> >
>> > Debug dump :
>> >
>> > 55 active threads
>> > Thread 10536 (IPC Client (47) connection to cldx-1139-1033/
>> > 172.25.6.69:9000 from hduser):
>> >   State: TIMED_WAITING
>> >   Blocked count: 1
>> >   Waited count: 2
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:706)
>> >     org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)
>> > Thread 10535 (IPC Client (47) connection to cldx-1139-1033/
>> > 172.25.6.69:60000 from hduser):
>> >   State: TIMED_WAITING
>> >   Blocked count: 3
>> >   Waited count: 4
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:534)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:579)
>> > Thread 10121 (1840929932@qtp-1863048490-7):
>> >   State: RUNNABLE
>> >   Blocked count: 77
>> >   Waited count: 415
>> >   Stack:
>> >     sun.management.ThreadImpl.getThreadInfo1(Native Method)
>> >     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
>> >     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
>> >
>> >
>> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
>> >
>> >
>> org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:203)
>> >
>> >
>> org.apache.hadoop.http.HttpServer$StackServlet.doGet(HttpServer.java:721)
>> >     javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>> >     javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> >
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>> >
>> >
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>> >
>> >
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>> >
>> >
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>> >
>> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>> >
>> >
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> >
>> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>> >
>> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> >
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>> >
>> >
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> >
>> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> >     org.mortbay.jetty.Server.handle(Server.java:326)
>> > Thread 1689 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-3):
>> >   State: WAITING
>> >   Blocked count: 2
>> >   Waited count: 4
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 260 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-2):
>> >   State: WAITING
>> >   Blocked count: 2
>> >   Waited count: 4
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 251
>> (MASTER_TABLE_OPERATIONS-cldx-1139-1033,60000,1365517046931-0):
>> >   State: WAITING
>> >   Blocked count: 25
>> >   Waited count: 31
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fce66ba
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 82 (cldx-1139-1033,60000,1365517046931-CatalogJanitor):
>> >   State: TIMED_WAITING
>> >   Blocked count: 5247
>> >   Waited count: 6944
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 81 (cldx-1139-1033,60000,1365517046931-BalancerChore):
>> >   State: TIMED_WAITING
>> >   Blocked count: 39
>> >   Waited count: 1736
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 80 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-1):
>> >   State: WAITING
>> >   Blocked count: 2
>> >   Waited count: 3
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 79 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-0):
>> >   State: WAITING
>> >   Blocked count: 2
>> >   Waited count: 3
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 77 (master-cldx-1139-1033,60000,1365517046931-EventThread):
>> >   State: WAITING
>> >   Blocked count: 1
>> >   Waited count: 5
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@10590940
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
>> > Thread 76
>> >
>> (master-cldx-1139-1033,60000,1365517046931-SendThread(cldx-1140-1034:2181)):
>> >   State: RUNNABLE
>> >   Blocked count: 40
>> >   Waited count: 0
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
>> >     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
>> > Thread 68 (cldx-1139-1033,60000,1365517046931.timeoutMonitor):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 52060
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 75 (Timer-0):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 17354
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.util.TimerThread.mainLoop(Timer.java:552)
>> >     java.util.TimerThread.run(Timer.java:505)
>> > Thread 74 (307539625@qtp-1863048490-1 - Acceptor0
>> > SelectChannelConnector@0.0.0.0:60010):
>> >   State: RUNNABLE
>> >   Blocked count: 92
>> >   Waited count: 1
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >
>> >
>> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>> >
>> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>> >
>> >
>> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>> >
>> >
>> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>> >
>> >
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> > Thread 71
>> (master-cldx-1139-1033,60000,1365517046931.archivedHFileCleaner):
>> >   State: TIMED_WAITING
>> >   Blocked count: 8718
>> >   Waited count: 17353
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 70 (master-cldx-1139-1033,60000,1365517046931.oldLogCleaner):
>> >   State: TIMED_WAITING
>> >   Blocked count: 8934
>> >   Waited count: 17353
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 69 (cldx-1139-1033,60000,1365517046931.timerUpdater):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 52061
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 45 (LeaseChecker):
>> >   State: TIMED_WAITING
>> >   Blocked count: 1
>> >   Waited count: 520519
>> >   Stack:
>> >     java.lang.Thread.sleep(Native Method)
>> >
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 43
>> > (cldx-1139-1033,60000,1365517046931.splitLogManagerTimeoutMonitor):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 520520
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 9 (master-cldx-1139-1033,60000,1365517046931):
>> >   State: TIMED_WAITING
>> >   Blocked count: 322
>> >   Waited count: 520958
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:55)
>> >     org.apache.hadoop.hbase.master.HMaster.loop(HMaster.java:489)
>> >     org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:387)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 40 (Timer thread for monitoring jvm):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 52062
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.util.TimerThread.mainLoop(Timer.java:552)
>> >     java.util.TimerThread.run(Timer.java:505)
>> > Thread 39 (Timer thread for monitoring hbase):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 52062
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.util.TimerThread.mainLoop(Timer.java:552)
>> >     java.util.TimerThread.run(Timer.java:505)
>> > Thread 38 (REPL IPC Server handler 2 on 60000):
>> >   State: WAITING
>> >   Blocked count: 0
>> >   Waited count: 1
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 37 (REPL IPC Server handler 1 on 60000):
>> >   State: WAITING
>> >   Blocked count: 0
>> >   Waited count: 1
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 36 (REPL IPC Server handler 0 on 60000):
>> >   State: WAITING
>> >   Blocked count: 0
>> >   Waited count: 1
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 35 (IPC Server handler 9 on 60000):
>> >   State: WAITING
>> >   Blocked count: 33
>> >   Waited count: 17376
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 34 (IPC Server handler 8 on 60000):
>> >   State: WAITING
>> >   Blocked count: 0
>> >   Waited count: 17343
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 33 (IPC Server handler 7 on 60000):
>> >   State: WAITING
>> >   Blocked count: 50
>> >   Waited count: 17393
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 32 (IPC Server handler 6 on 60000):
>> >   State: WAITING
>> >   Blocked count: 13
>> >   Waited count: 17355
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 31 (IPC Server handler 5 on 60000):
>> >   State: WAITING
>> >   Blocked count: 27
>> >   Waited count: 17368
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 30 (IPC Server handler 4 on 60000):
>> >   State: WAITING
>> >   Blocked count: 26
>> >   Waited count: 17370
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 29 (IPC Server handler 3 on 60000):
>> >   State: WAITING
>> >   Blocked count: 13
>> >   Waited count: 17357
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 28 (IPC Server handler 2 on 60000):
>> >   State: WAITING
>> >   Blocked count: 57
>> >   Waited count: 17401
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 27 (IPC Server handler 1 on 60000):
>> >   State: WAITING
>> >   Blocked count: 12
>> >   Waited count: 17356
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 26 (IPC Server handler 0 on 60000):
>> >   State: WAITING
>> >   Blocked count: 13
>> >   Waited count: 17357
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 10 (IPC Server listener on 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 17
>> >   Waited count: 0
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:636)
>> > Thread 23 (IPC Server Responder):
>> >   State: RUNNABLE
>> >   Blocked count: 0
>> >   Waited count: 0
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRunLoop(HBaseServer.java:825)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:808)
>> > Thread 25 (main-EventThread):
>> >   State: WAITING
>> >   Blocked count: 30
>> >   Waited count: 55
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f38d8fa
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
>> > Thread 24 (main-SendThread(cldx-1140-1034:2181)):
>> >   State: RUNNABLE
>> >   Blocked count: 50
>> >   Waited count: 0
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
>> >     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
>> > Thread 21 (Timer thread for monitoring rpc):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 52062
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.util.TimerThread.mainLoop(Timer.java:552)
>> >     java.util.TimerThread.run(Timer.java:505)
>> > Thread 20 (IPC Reader 9 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 1
>> >   Waited count: 1
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 19 (IPC Reader 8 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 1
>> >   Waited count: 1
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 18 (IPC Reader 7 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 17 (IPC Reader 6 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 16 (IPC Reader 5 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 15 (IPC Reader 4 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 14 (IPC Reader 3 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 13 (IPC Reader 2 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 12 (IPC Reader 1 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 3
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 11 (IPC Reader 0 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 1
>> >   Waited count: 1
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 5 (Signal Dispatcher):
>> >   State: RUNNABLE
>> >   Blocked count: 0
>> >   Waited count: 0
>> >   Stack:
>> > Thread 3 (Finalizer):
>> >   State: WAITING
>> >   Blocked count: 539
>> >   Waited count: 540
>> >   Waiting on java.lang.ref.ReferenceQueue$Lock@11499ff
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>> >     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>> >     java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
>> > Thread 2 (Reference Handler):
>> >   State: WAITING
>> >   Blocked count: 538
>> >   Waited count: 539
>> >   Waiting on java.lang.ref.Reference$Lock@6f8a34e6
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.lang.Object.wait(Object.java:503)
>> >     java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>> > Thread 1 (main):
>> >   State: WAITING
>> >   Blocked count: 16
>> >   Waited count: 17
>> >   Waiting on java.lang.Thread@3a246847
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.lang.Thread.join(Thread.java:1258)
>> >     java.lang.Thread.join(Thread.java:1332)
>> >     org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:86)
>> >
>> >
>> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:158)
>> >
>> >
>> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
>> >     org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> >
>> >
>> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
>> >     org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1944)
>> >
>> >
>> >
>> >
>> > Regards,
>> > Omkar Joshi
>> >
>> > -----Original Message-----
>> > From: Omkar Joshi [mailto:Omkar.Joshi@lntinfotech.com]
>> > Sent: Monday, April 15, 2013 9:12 AM
>> > To: user@hbase.apache.org
>> > Subject: Unable to connect from windows desktop to HBase
>> >
>> > Hi,
>> >
>> > I'm trying to connect from a Java client running on a Windows desktop
>> > machine to a remote HBase cluster running in the distributed mode(atop
>> > 2-node Hadoop cluster).
>> >
>> > 1.      On the master(namenode)node :
>> > hduser@cldx-1139-1033
>> :~/hadoop_ecosystem/apache_hbase/hbase_installation/hbase-0.94.6.1/bin$
>> > jps
>> > 9161 HMaster
>> > 4536 SecondaryNameNode
>> > 4368 DataNode
>> > 4645 JobTracker
>> > 8395 Jps
>> > 4813 TaskTracker
>> > 4179 NameNode
>> > 7717 Main
>> >
>> >
>> >
>> > 2.       On the slave(datanode)node :
>> >
>> >
>> > hduser@cldx-1140-1034:~$ jps
>> >
>> > 5677 HRegionServer
>> >
>> > 5559 HQuorumPeer
>> >
>> > 2634 TaskTracker
>> >
>> > 3260 Jps
>> >
>> > 2496 DataNode
>> >
>> >
>> >
>> > 3.       I also connected to the shell and created a hbase table and
>> also
>> > able to scan it :
>> >
>> >
>> > hbase(main):004:0> scan 'CUSTOMERS'
>> >
>> > ROW                              COLUMN+CELL
>> >
>> >  CUSTID12345                     column=CUSTOMER_INFO:EMAIL,
>> > timestamp=1365600369284, value=Omkar.Joshi@lntinfotech.com
>> >
>> >  CUSTID12345                     column=CUSTOMER_INFO:NAME,
>> > timestamp=1365600052104, value=Omkar Joshi
>> >
>> >  CUSTID614                       column=CUSTOMER_INFO:NAME,
>> > timestamp=1365601350972, value=Prachi Shah
>> >
>> > 2 row(s) in 0.8760 seconds
>> >
>> >
>> >
>> > 4.       The hbase-site.xml has the following configurations:
>> >
>> >
>> > <configuration>
>> >
>> > <property>
>> >
>> >     <name>hbase.zookeeper.quorum</name>
>> >
>> >     <value>cldx-1140-1034</value>
>> >
>> >     <description>The directory shared by RegionServers.
>> >
>> >     </description>
>> >
>> > </property>
>> >
>> > <property>
>> >
>> >     <name>hbase.zookeeper.property.dataDir</name>
>> >
>> >     <value>/home/hduser/hadoop_ecosystem/apache_hbase/zk_datadir</value>
>> >
>> >     <description>Property from ZooKeeper's config zoo.cfg.
>> >
>> >     The directory where the snapshot is stored.
>> >
>> >     </description>
>> >
>> > </property>
>> >
>> > <property>
>> >
>> >     <name>hbase.rootdir</name>
>> >
>> >     <value>hdfs://cldx-1139-1033:9000/hbase</value>
>> >
>> >     <description>The directory shared by RegionServers.</description>
>> >
>> > </property>
>> >
>> > <property>
>> >
>> >     <name>hbase.cluster.distributed</name>
>> >
>> >     <value>true</value>
>> >
>> >     <description>The mode the cluster will be in. Possible values are
>> > false: standalone and pseudo-distributed setups with managed Zookeeper
>> > true: fully-distributed with unmanaged Zookeeper Quorum (see
>> > hbase-env.sh)</description>
>> >
>> > </property>
>> >
>> > </configuration>
>> >
>> >
>> >
>> > 5.       The Hadoop's core-site.xml has the following settings :
>> >
>> >
>> > <configuration>
>> >
>> > <property>
>> >
>> > <name>fs.default.name</name>
>> >
>> > <value>hdfs://cldx-1139-1033:9000</value>
>> >
>> > </property>
>> >
>> > </configuration>
>> >
>> >
>> >
>> > 6.       My java client class is :
>> >
>> >
>> > package client.hbase;
>> >
>> >
>> >
>> > import java.io.IOException;
>> >
>> > import java.util.List;
>> >
>> >
>> >
>> > import org.apache.hadoop.conf.Configuration;
>> >
>> > import org.apache.hadoop.hbase.HBaseConfiguration;
>> >
>> > import org.apache.hadoop.hbase.HColumnDescriptor;
>> >
>> > import org.apache.hadoop.hbase.HTableDescriptor;
>> >
>> > import org.apache.hadoop.hbase.client.HBaseAdmin;
>> >
>> > import org.apache.hadoop.hbase.client.HTableInterface;
>> >
>> > import org.apache.hadoop.hbase.client.HTablePool;
>> >
>> > import org.apache.hadoop.hbase.client.Put;
>> >
>> > import org.apache.hadoop.hbase.client.Result;
>> >
>> > import org.apache.hadoop.hbase.client.ResultScanner;
>> >
>> > import org.apache.hadoop.hbase.client.Scan;
>> >
>> > import org.apache.hadoop.hbase.filter.CompareFilter;
>> >
>> > import org.apache.hadoop.hbase.filter.Filter;
>> >
>> > import org.apache.hadoop.hbase.filter.RegexStringComparator;
>> >
>> > import org.apache.hadoop.hbase.filter.ValueFilter;
>> >
>> > import org.apache.hadoop.hbase.util.Bytes;
>> >
>> >
>> >
>> > public class HBaseCRUD {
>> >
>> >
>> >
>> >     private static Configuration config;
>> >
>> >     private static HBaseAdmin hbaseAdmin;
>> >
>> >
>> >
>> >     private static HTablePool hTablePool;
>> >
>> >
>> >
>> >     static {
>> >
>> >         config = HBaseConfiguration.create();
>> >
>> >         config.set("hbase.zookeeper.quorum", "172.25.6.69");
>> >
>> >
>> >
>> >         hTablePool = new HTablePool(config, 2);
>> >
>> >     }
>> >
>> >
>> >
>> >     /**
>> >
>> >      * @param args
>> >
>> >      * @throws IOException
>> >
>> >      */
>> >
>> >     public static void main(String[] args) throws IOException {
>> >
>> >         // TODO Auto-generated method stub
>> >
>> >
>> >
>> >         HBaseCRUD hbaseCRUD = new HBaseCRUD();
>> >
>> >
>> >
>> >         /*hbaseCRUD.createTables("CUSTOMERS", "CUSTOMER_INFO");
>> >
>> >
>> >
>> >         hbaseCRUD.populateTableData("CUSTOMERS");*/
>> >
>> >
>> >
>> >         hbaseCRUD.scanTable("CUSTOMERS", "CUSTOMER_INFO", "EMAIL");
>> >
>> >     }
>> >
>> >
>> >
>> >     private void createTables(String tableName, String...
>> > columnFamilyNames)
>> >
>> >             throws IOException {
>> >
>> >         // TODO Auto-generated method stub
>> >
>> >
>> >
>> >         HTableDescriptor tableDesc = new HTableDescriptor(tableName);
>> >
>> >
>> >
>> >         if (!(columnFamilyNames == null || columnFamilyNames.length ==
>> 0))
>> > {
>> >
>> >
>> >
>> >             for (String columnFamilyName : columnFamilyNames) {
>> >
>> >
>> >
>> >                 HColumnDescriptor columnFamily = new HColumnDescriptor(
>> >
>> >                         columnFamilyName);
>> >
>> >
>> >
>> >                 tableDesc.addFamily(columnFamily);
>> >
>> >             }
>> >
>> >         }
>> >
>> >
>> >
>> >         hbaseAdmin.createTable(tableDesc);
>> >
>> >     }
>> >
>> >
>> >
>> >     private void populateTableData(String tableName) throws IOException
>> {
>> >
>> >         HTableInterface tbl =
>> > hTablePool.getTable(Bytes.toBytes(tableName));
>> >
>> >
>> >
>> >         List<Put> tblRows = getTableData(tableName);
>> >
>> >
>> >
>> >         tbl.close();
>> >
>> >     }
>> >
>> >
>> >
>> >     private List<Put> getTableData(String tableName) {
>> >
>> >         // TODO Auto-generated method stub
>> >
>> >
>> >
>> >         if (tableName == null || tableName.isEmpty())
>> >
>> >             return null;
>> >
>> >
>> >
>> >         /* Pull data from wherever located */
>> >
>> >
>> >
>> >         if (tableName.equalsIgnoreCase("CUSTOMERS")) {
>> >
>> >
>> >
>> >             /*
>> >
>> >              * Put p1 = new Put();
>> p1.add(Bytes.toBytes("CUSTOMER_INFO"),
>> >
>> >              * Bytes.toBytes("NAME"), value)
>> >
>> >              */
>> >
>> >         }
>> >
>> >
>> >
>> >         return null;
>> >
>> >     }
>> >
>> >
>> >
>> >     private void scanTable(String tableName, String columnFamilyName,
>> >
>> >             String... columnNames) throws IOException {
>> >
>> >
>> >
>> >         System.out.println("In HBaseCRUD.scanTable(...)");
>> >
>> >
>> >
>> >         Scan scan = new Scan();
>> >
>> >         if (!(columnNames == null || columnNames.length <= 0)) {
>> >
>> >
>> >
>> >             for (String columnName : columnNames) {
>> >
>> >
>> >
>> >                 scan.addColumn(Bytes.toBytes(columnFamilyName),
>> >
>> >                         Bytes.toBytes(columnName));
>> >
>> >             }
>> >
>> >
>> >
>> >             Filter filter = new
>> ValueFilter(CompareFilter.CompareOp.EQUAL,
>> >
>> >                     new RegexStringComparator("lntinfotech"));
>> >
>> >             scan.setFilter(filter);
>> >
>> >         }
>> >
>> >
>> >
>> >         HTableInterface tbl =
>> > hTablePool.getTable(Bytes.toBytes(tableName));
>> >
>> >         ResultScanner scanResults = tbl.getScanner(scan);
>> >
>> >
>> >
>> >         for (Result result : scanResults) {
>> >
>> >             System.out.println("The result is " + result);
>> >
>> >         }
>> >
>> >
>> >
>> >         tbl.close();
>> >
>> >     }
>> >
>> > }
>> >
>> >
>> >
>> > 7.       The exception I get is :
>> >
>> >
>> > In HBaseCRUD.scanTable(...)
>> >
>> > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>> >
>> > SLF4J: Defaulting to no-operation (NOP) logger implementation
>> >
>> > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>> further
>> > details.
>> >
>> > Apr 10, 2013 4:24:54 PM
>> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper <init>
>> >
>> > INFO: The identifier of this process is 3648@INFVA03351
>> >
>> > Apr 10, 2013 4:24:56 PM
>> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow
>> >
>> > WARNING: Possibly transient ZooKeeper exception:
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
>> >
>> > Apr 10, 2013 4:24:56 PM org.apache.hadoop.hbase.util.RetryCounter
>> > sleepUntilNextRetry
>> >
>> > INFO: Sleeping 2000ms before retry #1...
>> >
>> > Apr 10, 2013 4:24:58 PM
>> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow
>> >
>> > WARNING: Possibly transient ZooKeeper exception:
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
>> >
>> > Apr 10, 2013 4:24:58 PM org.apache.hadoop.hbase.util.RetryCounter
>> > sleepUntilNextRetry
>> >
>> > INFO: Sleeping 4000ms before retry #2...
>> >
>> >
>> > I have also made the master's entry in my local machine's host file.
>> What
>> > can be the error?
>> >
>> > Thanks and regards !
>> >
>> >
>> > ________________________________
>> > The contents of this e-mail and any attachment(s) may contain
>> confidential
>> > or privileged information for the intended recipient(s). Unintended
>> > recipients are prohibited from taking action on the basis of
>> information in
>> > this e-mail and using or disseminating the information, and must notify
>> the
>> > sender and delete it from their system. L&T Infotech will not accept
>> > responsibility or liability for the accuracy or completeness of, or the
>> > presence of any virus or disabling code in this e-mail"
>> >
>>
>
>

Re: Unable to connect from windows desktop to HBase

Posted by Azuryy Yu <az...@gmail.com>.
you need to put  protobuf-java-2.4.1.jar under the hbase_home/lib, then
don't include it in your client class path. but if you used hbase-0.95 or
hbase-0.96.1, protobuf jar is also needed on your client's classpath.



On Mon, Apr 15, 2013 at 6:37 PM, Azuryy Yu <az...@gmail.com> wrote:

>  config = HBaseConfiguration.create();
>
> this will read hbase-site.xml to get hmaster and zookeeper quorums, if
> not, then zookeeper quorums and hbase.master are all empty. so you cannot
> get connection.
>
>
>
>
> On Mon, Apr 15, 2013 at 6:29 PM, Omkar Joshi <Om...@lntinfotech.com>wrote:
>
>> Hi Azuryy,
>>
>> Thanks for the reply !
>>
>> It solved the issue but I'm not clear as to why this is required; also
>> had to include the Google's protobuf-java-2.4.1.jar whose purpose is
>> unclear to me.
>>
>> Regards,
>> Omkar Joshi
>>
>>
>> -----Original Message-----
>> From: Azuryy Yu [mailto:azuryyyu@gmail.com]
>> Sent: Monday, April 15, 2013 3:22 PM
>> To: user@hbase.apache.org
>> Subject: Re: Unable to connect from windows desktop to HBase
>>
>> you should put hbase-site.xml under your class path in your java client.
>>
>> then, don't need to specify zookeeper quorum in your code.
>>
>>
>> On Mon, Apr 15, 2013 at 5:35 PM, Omkar Joshi <Omkar.Joshi@lntinfotech.com
>> >wrote:
>>
>> > Attaching the dumps from the HBase UI :
>> >
>> > Debug dump :
>> >
>> > 55 active threads
>> > Thread 10536 (IPC Client (47) connection to cldx-1139-1033/
>> > 172.25.6.69:9000 from hduser):
>> >   State: TIMED_WAITING
>> >   Blocked count: 1
>> >   Waited count: 2
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:706)
>> >     org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)
>> > Thread 10535 (IPC Client (47) connection to cldx-1139-1033/
>> > 172.25.6.69:60000 from hduser):
>> >   State: TIMED_WAITING
>> >   Blocked count: 3
>> >   Waited count: 4
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:534)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:579)
>> > Thread 10121 (1840929932@qtp-1863048490-7):
>> >   State: RUNNABLE
>> >   Blocked count: 77
>> >   Waited count: 415
>> >   Stack:
>> >     sun.management.ThreadImpl.getThreadInfo1(Native Method)
>> >     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
>> >     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
>> >
>> >
>> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
>> >
>> >
>> org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:203)
>> >
>> >
>> org.apache.hadoop.http.HttpServer$StackServlet.doGet(HttpServer.java:721)
>> >     javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>> >     javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> >
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>> >
>> >
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>> >
>> >
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>> >
>> >
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>> >
>> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>> >
>> >
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> >
>> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>> >
>> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> >
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>> >
>> >
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> >
>> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> >     org.mortbay.jetty.Server.handle(Server.java:326)
>> > Thread 1689 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-3):
>> >   State: WAITING
>> >   Blocked count: 2
>> >   Waited count: 4
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 260 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-2):
>> >   State: WAITING
>> >   Blocked count: 2
>> >   Waited count: 4
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 251
>> (MASTER_TABLE_OPERATIONS-cldx-1139-1033,60000,1365517046931-0):
>> >   State: WAITING
>> >   Blocked count: 25
>> >   Waited count: 31
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fce66ba
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 82 (cldx-1139-1033,60000,1365517046931-CatalogJanitor):
>> >   State: TIMED_WAITING
>> >   Blocked count: 5247
>> >   Waited count: 6944
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 81 (cldx-1139-1033,60000,1365517046931-BalancerChore):
>> >   State: TIMED_WAITING
>> >   Blocked count: 39
>> >   Waited count: 1736
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 80 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-1):
>> >   State: WAITING
>> >   Blocked count: 2
>> >   Waited count: 3
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 79 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-0):
>> >   State: WAITING
>> >   Blocked count: 2
>> >   Waited count: 3
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 77 (master-cldx-1139-1033,60000,1365517046931-EventThread):
>> >   State: WAITING
>> >   Blocked count: 1
>> >   Waited count: 5
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@10590940
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
>> > Thread 76
>> >
>> (master-cldx-1139-1033,60000,1365517046931-SendThread(cldx-1140-1034:2181)):
>> >   State: RUNNABLE
>> >   Blocked count: 40
>> >   Waited count: 0
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
>> >     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
>> > Thread 68 (cldx-1139-1033,60000,1365517046931.timeoutMonitor):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 52060
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 75 (Timer-0):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 17354
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.util.TimerThread.mainLoop(Timer.java:552)
>> >     java.util.TimerThread.run(Timer.java:505)
>> > Thread 74 (307539625@qtp-1863048490-1 - Acceptor0
>> > SelectChannelConnector@0.0.0.0:60010):
>> >   State: RUNNABLE
>> >   Blocked count: 92
>> >   Waited count: 1
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >
>> >
>> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>> >
>> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>> >
>> >
>> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>> >
>> >
>> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>> >
>> >
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> > Thread 71
>> (master-cldx-1139-1033,60000,1365517046931.archivedHFileCleaner):
>> >   State: TIMED_WAITING
>> >   Blocked count: 8718
>> >   Waited count: 17353
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 70 (master-cldx-1139-1033,60000,1365517046931.oldLogCleaner):
>> >   State: TIMED_WAITING
>> >   Blocked count: 8934
>> >   Waited count: 17353
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 69 (cldx-1139-1033,60000,1365517046931.timerUpdater):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 52061
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 45 (LeaseChecker):
>> >   State: TIMED_WAITING
>> >   Blocked count: 1
>> >   Waited count: 520519
>> >   Stack:
>> >     java.lang.Thread.sleep(Native Method)
>> >
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 43
>> > (cldx-1139-1033,60000,1365517046931.splitLogManagerTimeoutMonitor):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 520520
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 9 (master-cldx-1139-1033,60000,1365517046931):
>> >   State: TIMED_WAITING
>> >   Blocked count: 322
>> >   Waited count: 520958
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:55)
>> >     org.apache.hadoop.hbase.master.HMaster.loop(HMaster.java:489)
>> >     org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:387)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 40 (Timer thread for monitoring jvm):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 52062
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.util.TimerThread.mainLoop(Timer.java:552)
>> >     java.util.TimerThread.run(Timer.java:505)
>> > Thread 39 (Timer thread for monitoring hbase):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 52062
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.util.TimerThread.mainLoop(Timer.java:552)
>> >     java.util.TimerThread.run(Timer.java:505)
>> > Thread 38 (REPL IPC Server handler 2 on 60000):
>> >   State: WAITING
>> >   Blocked count: 0
>> >   Waited count: 1
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 37 (REPL IPC Server handler 1 on 60000):
>> >   State: WAITING
>> >   Blocked count: 0
>> >   Waited count: 1
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 36 (REPL IPC Server handler 0 on 60000):
>> >   State: WAITING
>> >   Blocked count: 0
>> >   Waited count: 1
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 35 (IPC Server handler 9 on 60000):
>> >   State: WAITING
>> >   Blocked count: 33
>> >   Waited count: 17376
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 34 (IPC Server handler 8 on 60000):
>> >   State: WAITING
>> >   Blocked count: 0
>> >   Waited count: 17343
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 33 (IPC Server handler 7 on 60000):
>> >   State: WAITING
>> >   Blocked count: 50
>> >   Waited count: 17393
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 32 (IPC Server handler 6 on 60000):
>> >   State: WAITING
>> >   Blocked count: 13
>> >   Waited count: 17355
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 31 (IPC Server handler 5 on 60000):
>> >   State: WAITING
>> >   Blocked count: 27
>> >   Waited count: 17368
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 30 (IPC Server handler 4 on 60000):
>> >   State: WAITING
>> >   Blocked count: 26
>> >   Waited count: 17370
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 29 (IPC Server handler 3 on 60000):
>> >   State: WAITING
>> >   Blocked count: 13
>> >   Waited count: 17357
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 28 (IPC Server handler 2 on 60000):
>> >   State: WAITING
>> >   Blocked count: 57
>> >   Waited count: 17401
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 27 (IPC Server handler 1 on 60000):
>> >   State: WAITING
>> >   Blocked count: 12
>> >   Waited count: 17356
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 26 (IPC Server handler 0 on 60000):
>> >   State: WAITING
>> >   Blocked count: 13
>> >   Waited count: 17357
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
>> > Thread 10 (IPC Server listener on 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 17
>> >   Waited count: 0
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:636)
>> > Thread 23 (IPC Server Responder):
>> >   State: RUNNABLE
>> >   Blocked count: 0
>> >   Waited count: 0
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRunLoop(HBaseServer.java:825)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:808)
>> > Thread 25 (main-EventThread):
>> >   State: WAITING
>> >   Blocked count: 30
>> >   Waited count: 55
>> >   Waiting on
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f38d8fa
>> >   Stack:
>> >     sun.misc.Unsafe.park(Native Method)
>> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>> >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> >     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
>> > Thread 24 (main-SendThread(cldx-1140-1034:2181)):
>> >   State: RUNNABLE
>> >   Blocked count: 50
>> >   Waited count: 0
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
>> >     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
>> > Thread 21 (Timer thread for monitoring rpc):
>> >   State: TIMED_WAITING
>> >   Blocked count: 0
>> >   Waited count: 52062
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.util.TimerThread.mainLoop(Timer.java:552)
>> >     java.util.TimerThread.run(Timer.java:505)
>> > Thread 20 (IPC Reader 9 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 1
>> >   Waited count: 1
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 19 (IPC Reader 8 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 1
>> >   Waited count: 1
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 18 (IPC Reader 7 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 17 (IPC Reader 6 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 16 (IPC Reader 5 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 15 (IPC Reader 4 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 14 (IPC Reader 3 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 13 (IPC Reader 2 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 2
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 12 (IPC Reader 1 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 3
>> >   Waited count: 2
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 11 (IPC Reader 0 on port 60000):
>> >   State: RUNNABLE
>> >   Blocked count: 1
>> >   Waited count: 1
>> >   Stack:
>> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>> >
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >     java.lang.Thread.run(Thread.java:722)
>> > Thread 5 (Signal Dispatcher):
>> >   State: RUNNABLE
>> >   Blocked count: 0
>> >   Waited count: 0
>> >   Stack:
>> > Thread 3 (Finalizer):
>> >   State: WAITING
>> >   Blocked count: 539
>> >   Waited count: 540
>> >   Waiting on java.lang.ref.ReferenceQueue$Lock@11499ff
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>> >     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>> >     java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
>> > Thread 2 (Reference Handler):
>> >   State: WAITING
>> >   Blocked count: 538
>> >   Waited count: 539
>> >   Waiting on java.lang.ref.Reference$Lock@6f8a34e6
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.lang.Object.wait(Object.java:503)
>> >     java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>> > Thread 1 (main):
>> >   State: WAITING
>> >   Blocked count: 16
>> >   Waited count: 17
>> >   Waiting on java.lang.Thread@3a246847
>> >   Stack:
>> >     java.lang.Object.wait(Native Method)
>> >     java.lang.Thread.join(Thread.java:1258)
>> >     java.lang.Thread.join(Thread.java:1332)
>> >     org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:86)
>> >
>> >
>> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:158)
>> >
>> >
>> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
>> >     org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> >
>> >
>> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
>> >     org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1944)
>> >
>> >
>> >
>> >
>> > Regards,
>> > Omkar Joshi
>> >
>> > -----Original Message-----
>> > From: Omkar Joshi [mailto:Omkar.Joshi@lntinfotech.com]
>> > Sent: Monday, April 15, 2013 9:12 AM
>> > To: user@hbase.apache.org
>> > Subject: Unable to connect from windows desktop to HBase
>> >
>> > Hi,
>> >
>> > I'm trying to connect from a Java client running on a Windows desktop
>> > machine to a remote HBase cluster running in the distributed mode(atop
>> > 2-node Hadoop cluster).
>> >
>> > 1.      On the master(namenode)node :
>> > hduser@cldx-1139-1033
>> :~/hadoop_ecosystem/apache_hbase/hbase_installation/hbase-0.94.6.1/bin$
>> > jps
>> > 9161 HMaster
>> > 4536 SecondaryNameNode
>> > 4368 DataNode
>> > 4645 JobTracker
>> > 8395 Jps
>> > 4813 TaskTracker
>> > 4179 NameNode
>> > 7717 Main
>> >
>> >
>> >
>> > 2.       On the slave(datanode)node :
>> >
>> >
>> > hduser@cldx-1140-1034:~$ jps
>> >
>> > 5677 HRegionServer
>> >
>> > 5559 HQuorumPeer
>> >
>> > 2634 TaskTracker
>> >
>> > 3260 Jps
>> >
>> > 2496 DataNode
>> >
>> >
>> >
>> > 3.       I also connected to the shell and created a hbase table and
>> also
>> > able to scan it :
>> >
>> >
>> > hbase(main):004:0> scan 'CUSTOMERS'
>> >
>> > ROW                              COLUMN+CELL
>> >
>> >  CUSTID12345                     column=CUSTOMER_INFO:EMAIL,
>> > timestamp=1365600369284, value=Omkar.Joshi@lntinfotech.com
>> >
>> >  CUSTID12345                     column=CUSTOMER_INFO:NAME,
>> > timestamp=1365600052104, value=Omkar Joshi
>> >
>> >  CUSTID614                       column=CUSTOMER_INFO:NAME,
>> > timestamp=1365601350972, value=Prachi Shah
>> >
>> > 2 row(s) in 0.8760 seconds
>> >
>> >
>> >
>> > 4.       The hbase-site.xml has the following configurations:
>> >
>> >
>> > <configuration>
>> >
>> > <property>
>> >
>> >     <name>hbase.zookeeper.quorum</name>
>> >
>> >     <value>cldx-1140-1034</value>
>> >
>> >     <description>The directory shared by RegionServers.
>> >
>> >     </description>
>> >
>> > </property>
>> >
>> > <property>
>> >
>> >     <name>hbase.zookeeper.property.dataDir</name>
>> >
>> >     <value>/home/hduser/hadoop_ecosystem/apache_hbase/zk_datadir</value>
>> >
>> >     <description>Property from ZooKeeper's config zoo.cfg.
>> >
>> >     The directory where the snapshot is stored.
>> >
>> >     </description>
>> >
>> > </property>
>> >
>> > <property>
>> >
>> >     <name>hbase.rootdir</name>
>> >
>> >     <value>hdfs://cldx-1139-1033:9000/hbase</value>
>> >
>> >     <description>The directory shared by RegionServers.</description>
>> >
>> > </property>
>> >
>> > <property>
>> >
>> >     <name>hbase.cluster.distributed</name>
>> >
>> >     <value>true</value>
>> >
>> >     <description>The mode the cluster will be in. Possible values are
>> > false: standalone and pseudo-distributed setups with managed Zookeeper
>> > true: fully-distributed with unmanaged Zookeeper Quorum (see
>> > hbase-env.sh)</description>
>> >
>> > </property>
>> >
>> > </configuration>
>> >
>> >
>> >
>> > 5.       The Hadoop's core-site.xml has the following settings :
>> >
>> >
>> > <configuration>
>> >
>> > <property>
>> >
>> > <name>fs.default.name</name>
>> >
>> > <value>hdfs://cldx-1139-1033:9000</value>
>> >
>> > </property>
>> >
>> > </configuration>
>> >
>> >
>> >
>> > 6.       My java client class is :
>> >
>> >
>> > package client.hbase;
>> >
>> >
>> >
>> > import java.io.IOException;
>> >
>> > import java.util.List;
>> >
>> >
>> >
>> > import org.apache.hadoop.conf.Configuration;
>> >
>> > import org.apache.hadoop.hbase.HBaseConfiguration;
>> >
>> > import org.apache.hadoop.hbase.HColumnDescriptor;
>> >
>> > import org.apache.hadoop.hbase.HTableDescriptor;
>> >
>> > import org.apache.hadoop.hbase.client.HBaseAdmin;
>> >
>> > import org.apache.hadoop.hbase.client.HTableInterface;
>> >
>> > import org.apache.hadoop.hbase.client.HTablePool;
>> >
>> > import org.apache.hadoop.hbase.client.Put;
>> >
>> > import org.apache.hadoop.hbase.client.Result;
>> >
>> > import org.apache.hadoop.hbase.client.ResultScanner;
>> >
>> > import org.apache.hadoop.hbase.client.Scan;
>> >
>> > import org.apache.hadoop.hbase.filter.CompareFilter;
>> >
>> > import org.apache.hadoop.hbase.filter.Filter;
>> >
>> > import org.apache.hadoop.hbase.filter.RegexStringComparator;
>> >
>> > import org.apache.hadoop.hbase.filter.ValueFilter;
>> >
>> > import org.apache.hadoop.hbase.util.Bytes;
>> >
>> >
>> >
>> > public class HBaseCRUD {
>> >
>> >
>> >
>> >     private static Configuration config;
>> >
>> >     private static HBaseAdmin hbaseAdmin;
>> >
>> >
>> >
>> >     private static HTablePool hTablePool;
>> >
>> >
>> >
>> >     static {
>> >
>> >         config = HBaseConfiguration.create();
>> >
>> >         config.set("hbase.zookeeper.quorum", "172.25.6.69");
>> >
>> >
>> >
>> >         hTablePool = new HTablePool(config, 2);
>> >
>> >     }
>> >
>> >
>> >
>> >     /**
>> >
>> >      * @param args
>> >
>> >      * @throws IOException
>> >
>> >      */
>> >
>> >     public static void main(String[] args) throws IOException {
>> >
>> >         // TODO Auto-generated method stub
>> >
>> >
>> >
>> >         HBaseCRUD hbaseCRUD = new HBaseCRUD();
>> >
>> >
>> >
>> >         /*hbaseCRUD.createTables("CUSTOMERS", "CUSTOMER_INFO");
>> >
>> >
>> >
>> >         hbaseCRUD.populateTableData("CUSTOMERS");*/
>> >
>> >
>> >
>> >         hbaseCRUD.scanTable("CUSTOMERS", "CUSTOMER_INFO", "EMAIL");
>> >
>> >     }
>> >
>> >
>> >
>> >     private void createTables(String tableName, String...
>> > columnFamilyNames)
>> >
>> >             throws IOException {
>> >
>> >         // TODO Auto-generated method stub
>> >
>> >
>> >
>> >         HTableDescriptor tableDesc = new HTableDescriptor(tableName);
>> >
>> >
>> >
>> >         if (!(columnFamilyNames == null || columnFamilyNames.length ==
>> 0))
>> > {
>> >
>> >
>> >
>> >             for (String columnFamilyName : columnFamilyNames) {
>> >
>> >
>> >
>> >                 HColumnDescriptor columnFamily = new HColumnDescriptor(
>> >
>> >                         columnFamilyName);
>> >
>> >
>> >
>> >                 tableDesc.addFamily(columnFamily);
>> >
>> >             }
>> >
>> >         }
>> >
>> >
>> >
>> >         hbaseAdmin.createTable(tableDesc);
>> >
>> >     }
>> >
>> >
>> >
>> >     private void populateTableData(String tableName) throws IOException
>> {
>> >
>> >         HTableInterface tbl =
>> > hTablePool.getTable(Bytes.toBytes(tableName));
>> >
>> >
>> >
>> >         List<Put> tblRows = getTableData(tableName);
>> >
>> >
>> >
>> >         tbl.close();
>> >
>> >     }
>> >
>> >
>> >
>> >     private List<Put> getTableData(String tableName) {
>> >
>> >         // TODO Auto-generated method stub
>> >
>> >
>> >
>> >         if (tableName == null || tableName.isEmpty())
>> >
>> >             return null;
>> >
>> >
>> >
>> >         /* Pull data from wherever located */
>> >
>> >
>> >
>> >         if (tableName.equalsIgnoreCase("CUSTOMERS")) {
>> >
>> >
>> >
>> >             /*
>> >
>> >              * Put p1 = new Put();
>> p1.add(Bytes.toBytes("CUSTOMER_INFO"),
>> >
>> >              * Bytes.toBytes("NAME"), value)
>> >
>> >              */
>> >
>> >         }
>> >
>> >
>> >
>> >         return null;
>> >
>> >     }
>> >
>> >
>> >
>> >     private void scanTable(String tableName, String columnFamilyName,
>> >
>> >             String... columnNames) throws IOException {
>> >
>> >
>> >
>> >         System.out.println("In HBaseCRUD.scanTable(...)");
>> >
>> >
>> >
>> >         Scan scan = new Scan();
>> >
>> >         if (!(columnNames == null || columnNames.length <= 0)) {
>> >
>> >
>> >
>> >             for (String columnName : columnNames) {
>> >
>> >
>> >
>> >                 scan.addColumn(Bytes.toBytes(columnFamilyName),
>> >
>> >                         Bytes.toBytes(columnName));
>> >
>> >             }
>> >
>> >
>> >
>> >             Filter filter = new
>> ValueFilter(CompareFilter.CompareOp.EQUAL,
>> >
>> >                     new RegexStringComparator("lntinfotech"));
>> >
>> >             scan.setFilter(filter);
>> >
>> >         }
>> >
>> >
>> >
>> >         HTableInterface tbl =
>> > hTablePool.getTable(Bytes.toBytes(tableName));
>> >
>> >         ResultScanner scanResults = tbl.getScanner(scan);
>> >
>> >
>> >
>> >         for (Result result : scanResults) {
>> >
>> >             System.out.println("The result is " + result);
>> >
>> >         }
>> >
>> >
>> >
>> >         tbl.close();
>> >
>> >     }
>> >
>> > }
>> >
>> >
>> >
>> > 7.       The exception I get is :
>> >
>> >
>> > In HBaseCRUD.scanTable(...)
>> >
>> > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>> >
>> > SLF4J: Defaulting to no-operation (NOP) logger implementation
>> >
>> > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>> further
>> > details.
>> >
>> > Apr 10, 2013 4:24:54 PM
>> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper <init>
>> >
>> > INFO: The identifier of this process is 3648@INFVA03351
>> >
>> > Apr 10, 2013 4:24:56 PM
>> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow
>> >
>> > WARNING: Possibly transient ZooKeeper exception:
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
>> >
>> > Apr 10, 2013 4:24:56 PM org.apache.hadoop.hbase.util.RetryCounter
>> > sleepUntilNextRetry
>> >
>> > INFO: Sleeping 2000ms before retry #1...
>> >
>> > Apr 10, 2013 4:24:58 PM
>> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow
>> >
>> > WARNING: Possibly transient ZooKeeper exception:
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
>> >
>> > Apr 10, 2013 4:24:58 PM org.apache.hadoop.hbase.util.RetryCounter
>> > sleepUntilNextRetry
>> >
>> > INFO: Sleeping 4000ms before retry #2...
>> >
>> >
>> > I have also made the master's entry in my local machine's host file.
>> What
>> > can be the error?
>> >
>> > Thanks and regards !
>> >
>> >
>> > ________________________________
>> > The contents of this e-mail and any attachment(s) may contain
>> confidential
>> > or privileged information for the intended recipient(s). Unintended
>> > recipients are prohibited from taking action on the basis of
>> information in
>> > this e-mail and using or disseminating the information, and must notify
>> the
>> > sender and delete it from their system. L&T Infotech will not accept
>> > responsibility or liability for the accuracy or completeness of, or the
>> > presence of any virus or disabling code in this e-mail"
>> >
>>
>
>

Re: Unable to connect from windows desktop to HBase

Posted by Azuryy Yu <az...@gmail.com>.
 config = HBaseConfiguration.create();

this will read hbase-site.xml to get hmaster and zookeeper quorums, if not,
then zookeeper quorums and hbase.master are all empty. so you cannot get
connection.




On Mon, Apr 15, 2013 at 6:29 PM, Omkar Joshi <Om...@lntinfotech.com>wrote:

> Hi Azuryy,
>
> Thanks for the reply !
>
> It solved the issue but I'm not clear as to why this is required; also had
> to include the Google's protobuf-java-2.4.1.jar whose purpose is unclear to
> me.
>
> Regards,
> Omkar Joshi
>
>
> -----Original Message-----
> From: Azuryy Yu [mailto:azuryyyu@gmail.com]
> Sent: Monday, April 15, 2013 3:22 PM
> To: user@hbase.apache.org
> Subject: Re: Unable to connect from windows desktop to HBase
>
> you should put hbase-site.xml under your class path in your java client.
>
> then, don't need to specify zookeeper quorum in your code.
>
>
> On Mon, Apr 15, 2013 at 5:35 PM, Omkar Joshi <Omkar.Joshi@lntinfotech.com
> >wrote:
>
> > Attaching the dumps from the HBase UI :
> >
> > Debug dump :
> >
> > 55 active threads
> > Thread 10536 (IPC Client (47) connection to cldx-1139-1033/
> > 172.25.6.69:9000 from hduser):
> >   State: TIMED_WAITING
> >   Blocked count: 1
> >   Waited count: 2
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:706)
> >     org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)
> > Thread 10535 (IPC Client (47) connection to cldx-1139-1033/
> > 172.25.6.69:60000 from hduser):
> >   State: TIMED_WAITING
> >   Blocked count: 3
> >   Waited count: 4
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:534)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:579)
> > Thread 10121 (1840929932@qtp-1863048490-7):
> >   State: RUNNABLE
> >   Blocked count: 77
> >   Waited count: 415
> >   Stack:
> >     sun.management.ThreadImpl.getThreadInfo1(Native Method)
> >     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
> >     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
> >
> >
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> >
> >
> org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:203)
> >
> > org.apache.hadoop.http.HttpServer$StackServlet.doGet(HttpServer.java:721)
> >     javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> >     javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> >
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> >
> >
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
> >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> >
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> >
> >
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> >
> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> >
> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> >     org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> >
> >
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> >
> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> >     org.mortbay.jetty.Server.handle(Server.java:326)
> > Thread 1689 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-3):
> >   State: WAITING
> >   Blocked count: 2
> >   Waited count: 4
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 260 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-2):
> >   State: WAITING
> >   Blocked count: 2
> >   Waited count: 4
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 251
> (MASTER_TABLE_OPERATIONS-cldx-1139-1033,60000,1365517046931-0):
> >   State: WAITING
> >   Blocked count: 25
> >   Waited count: 31
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fce66ba
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 82 (cldx-1139-1033,60000,1365517046931-CatalogJanitor):
> >   State: TIMED_WAITING
> >   Blocked count: 5247
> >   Waited count: 6944
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 81 (cldx-1139-1033,60000,1365517046931-BalancerChore):
> >   State: TIMED_WAITING
> >   Blocked count: 39
> >   Waited count: 1736
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 80 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-1):
> >   State: WAITING
> >   Blocked count: 2
> >   Waited count: 3
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 79 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-0):
> >   State: WAITING
> >   Blocked count: 2
> >   Waited count: 3
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 77 (master-cldx-1139-1033,60000,1365517046931-EventThread):
> >   State: WAITING
> >   Blocked count: 1
> >   Waited count: 5
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@10590940
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
> > Thread 76
> >
> (master-cldx-1139-1033,60000,1365517046931-SendThread(cldx-1140-1034:2181)):
> >   State: RUNNABLE
> >   Blocked count: 40
> >   Waited count: 0
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
> >     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> > Thread 68 (cldx-1139-1033,60000,1365517046931.timeoutMonitor):
> >   State: TIMED_WAITING
> >   Blocked count: 0
> >   Waited count: 52060
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 75 (Timer-0):
> >   State: TIMED_WAITING
> >   Blocked count: 0
> >   Waited count: 17354
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     java.util.TimerThread.mainLoop(Timer.java:552)
> >     java.util.TimerThread.run(Timer.java:505)
> > Thread 74 (307539625@qtp-1863048490-1 - Acceptor0
> > SelectChannelConnector@0.0.0.0:60010):
> >   State: RUNNABLE
> >   Blocked count: 92
> >   Waited count: 1
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >
> >
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> >     org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> >
> >
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> >
> >
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> >
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> > Thread 71
> (master-cldx-1139-1033,60000,1365517046931.archivedHFileCleaner):
> >   State: TIMED_WAITING
> >   Blocked count: 8718
> >   Waited count: 17353
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 70 (master-cldx-1139-1033,60000,1365517046931.oldLogCleaner):
> >   State: TIMED_WAITING
> >   Blocked count: 8934
> >   Waited count: 17353
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 69 (cldx-1139-1033,60000,1365517046931.timerUpdater):
> >   State: TIMED_WAITING
> >   Blocked count: 0
> >   Waited count: 52061
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 45 (LeaseChecker):
> >   State: TIMED_WAITING
> >   Blocked count: 1
> >   Waited count: 520519
> >   Stack:
> >     java.lang.Thread.sleep(Native Method)
> >
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 43
> > (cldx-1139-1033,60000,1365517046931.splitLogManagerTimeoutMonitor):
> >   State: TIMED_WAITING
> >   Blocked count: 0
> >   Waited count: 520520
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
> >     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 9 (master-cldx-1139-1033,60000,1365517046931):
> >   State: TIMED_WAITING
> >   Blocked count: 322
> >   Waited count: 520958
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
> >     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:55)
> >     org.apache.hadoop.hbase.master.HMaster.loop(HMaster.java:489)
> >     org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:387)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 40 (Timer thread for monitoring jvm):
> >   State: TIMED_WAITING
> >   Blocked count: 0
> >   Waited count: 52062
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     java.util.TimerThread.mainLoop(Timer.java:552)
> >     java.util.TimerThread.run(Timer.java:505)
> > Thread 39 (Timer thread for monitoring hbase):
> >   State: TIMED_WAITING
> >   Blocked count: 0
> >   Waited count: 52062
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     java.util.TimerThread.mainLoop(Timer.java:552)
> >     java.util.TimerThread.run(Timer.java:505)
> > Thread 38 (REPL IPC Server handler 2 on 60000):
> >   State: WAITING
> >   Blocked count: 0
> >   Waited count: 1
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 37 (REPL IPC Server handler 1 on 60000):
> >   State: WAITING
> >   Blocked count: 0
> >   Waited count: 1
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 36 (REPL IPC Server handler 0 on 60000):
> >   State: WAITING
> >   Blocked count: 0
> >   Waited count: 1
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 35 (IPC Server handler 9 on 60000):
> >   State: WAITING
> >   Blocked count: 33
> >   Waited count: 17376
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 34 (IPC Server handler 8 on 60000):
> >   State: WAITING
> >   Blocked count: 0
> >   Waited count: 17343
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 33 (IPC Server handler 7 on 60000):
> >   State: WAITING
> >   Blocked count: 50
> >   Waited count: 17393
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 32 (IPC Server handler 6 on 60000):
> >   State: WAITING
> >   Blocked count: 13
> >   Waited count: 17355
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 31 (IPC Server handler 5 on 60000):
> >   State: WAITING
> >   Blocked count: 27
> >   Waited count: 17368
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 30 (IPC Server handler 4 on 60000):
> >   State: WAITING
> >   Blocked count: 26
> >   Waited count: 17370
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 29 (IPC Server handler 3 on 60000):
> >   State: WAITING
> >   Blocked count: 13
> >   Waited count: 17357
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 28 (IPC Server handler 2 on 60000):
> >   State: WAITING
> >   Blocked count: 57
> >   Waited count: 17401
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 27 (IPC Server handler 1 on 60000):
> >   State: WAITING
> >   Blocked count: 12
> >   Waited count: 17356
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 26 (IPC Server handler 0 on 60000):
> >   State: WAITING
> >   Blocked count: 13
> >   Waited count: 17357
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> > Thread 10 (IPC Server listener on 60000):
> >   State: RUNNABLE
> >   Blocked count: 17
> >   Waited count: 0
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:636)
> > Thread 23 (IPC Server Responder):
> >   State: RUNNABLE
> >   Blocked count: 0
> >   Waited count: 0
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRunLoop(HBaseServer.java:825)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:808)
> > Thread 25 (main-EventThread):
> >   State: WAITING
> >   Blocked count: 30
> >   Waited count: 55
> >   Waiting on
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f38d8fa
> >   Stack:
> >     sun.misc.Unsafe.park(Native Method)
> >     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
> > Thread 24 (main-SendThread(cldx-1140-1034:2181)):
> >   State: RUNNABLE
> >   Blocked count: 50
> >   Waited count: 0
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
> >     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> > Thread 21 (Timer thread for monitoring rpc):
> >   State: TIMED_WAITING
> >   Blocked count: 0
> >   Waited count: 52062
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     java.util.TimerThread.mainLoop(Timer.java:552)
> >     java.util.TimerThread.run(Timer.java:505)
> > Thread 20 (IPC Reader 9 on port 60000):
> >   State: RUNNABLE
> >   Blocked count: 1
> >   Waited count: 1
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 19 (IPC Reader 8 on port 60000):
> >   State: RUNNABLE
> >   Blocked count: 1
> >   Waited count: 1
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 18 (IPC Reader 7 on port 60000):
> >   State: RUNNABLE
> >   Blocked count: 2
> >   Waited count: 2
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 17 (IPC Reader 6 on port 60000):
> >   State: RUNNABLE
> >   Blocked count: 2
> >   Waited count: 2
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 16 (IPC Reader 5 on port 60000):
> >   State: RUNNABLE
> >   Blocked count: 2
> >   Waited count: 2
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 15 (IPC Reader 4 on port 60000):
> >   State: RUNNABLE
> >   Blocked count: 2
> >   Waited count: 2
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 14 (IPC Reader 3 on port 60000):
> >   State: RUNNABLE
> >   Blocked count: 2
> >   Waited count: 2
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 13 (IPC Reader 2 on port 60000):
> >   State: RUNNABLE
> >   Blocked count: 2
> >   Waited count: 2
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 12 (IPC Reader 1 on port 60000):
> >   State: RUNNABLE
> >   Blocked count: 3
> >   Waited count: 2
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 11 (IPC Reader 0 on port 60000):
> >   State: RUNNABLE
> >   Blocked count: 1
> >   Waited count: 1
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     java.lang.Thread.run(Thread.java:722)
> > Thread 5 (Signal Dispatcher):
> >   State: RUNNABLE
> >   Blocked count: 0
> >   Waited count: 0
> >   Stack:
> > Thread 3 (Finalizer):
> >   State: WAITING
> >   Blocked count: 539
> >   Waited count: 540
> >   Waiting on java.lang.ref.ReferenceQueue$Lock@11499ff
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> >     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> >     java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
> > Thread 2 (Reference Handler):
> >   State: WAITING
> >   Blocked count: 538
> >   Waited count: 539
> >   Waiting on java.lang.ref.Reference$Lock@6f8a34e6
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     java.lang.Object.wait(Object.java:503)
> >     java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> > Thread 1 (main):
> >   State: WAITING
> >   Blocked count: 16
> >   Waited count: 17
> >   Waiting on java.lang.Thread@3a246847
> >   Stack:
> >     java.lang.Object.wait(Native Method)
> >     java.lang.Thread.join(Thread.java:1258)
> >     java.lang.Thread.join(Thread.java:1332)
> >     org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:86)
> >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:158)
> >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
> >     org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >
> >
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
> >     org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1944)
> >
> >
> >
> >
> > Regards,
> > Omkar Joshi
> >
> > -----Original Message-----
> > From: Omkar Joshi [mailto:Omkar.Joshi@lntinfotech.com]
> > Sent: Monday, April 15, 2013 9:12 AM
> > To: user@hbase.apache.org
> > Subject: Unable to connect from windows desktop to HBase
> >
> > Hi,
> >
> > I'm trying to connect from a Java client running on a Windows desktop
> > machine to a remote HBase cluster running in the distributed mode(atop
> > 2-node Hadoop cluster).
> >
> > 1.      On the master(namenode)node :
> > hduser@cldx-1139-1033
> :~/hadoop_ecosystem/apache_hbase/hbase_installation/hbase-0.94.6.1/bin$
> > jps
> > 9161 HMaster
> > 4536 SecondaryNameNode
> > 4368 DataNode
> > 4645 JobTracker
> > 8395 Jps
> > 4813 TaskTracker
> > 4179 NameNode
> > 7717 Main
> >
> >
> >
> > 2.       On the slave(datanode)node :
> >
> >
> > hduser@cldx-1140-1034:~$ jps
> >
> > 5677 HRegionServer
> >
> > 5559 HQuorumPeer
> >
> > 2634 TaskTracker
> >
> > 3260 Jps
> >
> > 2496 DataNode
> >
> >
> >
> > 3.       I also connected to the shell and created a hbase table and also
> > able to scan it :
> >
> >
> > hbase(main):004:0> scan 'CUSTOMERS'
> >
> > ROW                              COLUMN+CELL
> >
> >  CUSTID12345                     column=CUSTOMER_INFO:EMAIL,
> > timestamp=1365600369284, value=Omkar.Joshi@lntinfotech.com
> >
> >  CUSTID12345                     column=CUSTOMER_INFO:NAME,
> > timestamp=1365600052104, value=Omkar Joshi
> >
> >  CUSTID614                       column=CUSTOMER_INFO:NAME,
> > timestamp=1365601350972, value=Prachi Shah
> >
> > 2 row(s) in 0.8760 seconds
> >
> >
> >
> > 4.       The hbase-site.xml has the following configurations:
> >
> >
> > <configuration>
> >
> > <property>
> >
> >     <name>hbase.zookeeper.quorum</name>
> >
> >     <value>cldx-1140-1034</value>
> >
> >     <description>The directory shared by RegionServers.
> >
> >     </description>
> >
> > </property>
> >
> > <property>
> >
> >     <name>hbase.zookeeper.property.dataDir</name>
> >
> >     <value>/home/hduser/hadoop_ecosystem/apache_hbase/zk_datadir</value>
> >
> >     <description>Property from ZooKeeper's config zoo.cfg.
> >
> >     The directory where the snapshot is stored.
> >
> >     </description>
> >
> > </property>
> >
> > <property>
> >
> >     <name>hbase.rootdir</name>
> >
> >     <value>hdfs://cldx-1139-1033:9000/hbase</value>
> >
> >     <description>The directory shared by RegionServers.</description>
> >
> > </property>
> >
> > <property>
> >
> >     <name>hbase.cluster.distributed</name>
> >
> >     <value>true</value>
> >
> >     <description>The mode the cluster will be in. Possible values are
> > false: standalone and pseudo-distributed setups with managed Zookeeper
> > true: fully-distributed with unmanaged Zookeeper Quorum (see
> > hbase-env.sh)</description>
> >
> > </property>
> >
> > </configuration>
> >
> >
> >
> > 5.       The Hadoop's core-site.xml has the following settings :
> >
> >
> > <configuration>
> >
> > <property>
> >
> > <name>fs.default.name</name>
> >
> > <value>hdfs://cldx-1139-1033:9000</value>
> >
> > </property>
> >
> > </configuration>
> >
> >
> >
> > 6.       My java client class is :
> >
> >
> > package client.hbase;
> >
> >
> >
> > import java.io.IOException;
> >
> > import java.util.List;
> >
> >
> >
> > import org.apache.hadoop.conf.Configuration;
> >
> > import org.apache.hadoop.hbase.HBaseConfiguration;
> >
> > import org.apache.hadoop.hbase.HColumnDescriptor;
> >
> > import org.apache.hadoop.hbase.HTableDescriptor;
> >
> > import org.apache.hadoop.hbase.client.HBaseAdmin;
> >
> > import org.apache.hadoop.hbase.client.HTableInterface;
> >
> > import org.apache.hadoop.hbase.client.HTablePool;
> >
> > import org.apache.hadoop.hbase.client.Put;
> >
> > import org.apache.hadoop.hbase.client.Result;
> >
> > import org.apache.hadoop.hbase.client.ResultScanner;
> >
> > import org.apache.hadoop.hbase.client.Scan;
> >
> > import org.apache.hadoop.hbase.filter.CompareFilter;
> >
> > import org.apache.hadoop.hbase.filter.Filter;
> >
> > import org.apache.hadoop.hbase.filter.RegexStringComparator;
> >
> > import org.apache.hadoop.hbase.filter.ValueFilter;
> >
> > import org.apache.hadoop.hbase.util.Bytes;
> >
> >
> >
> > public class HBaseCRUD {
> >
> >
> >
> >     private static Configuration config;
> >
> >     private static HBaseAdmin hbaseAdmin;
> >
> >
> >
> >     private static HTablePool hTablePool;
> >
> >
> >
> >     static {
> >
> >         config = HBaseConfiguration.create();
> >
> >         config.set("hbase.zookeeper.quorum", "172.25.6.69");
> >
> >
> >
> >         hTablePool = new HTablePool(config, 2);
> >
> >     }
> >
> >
> >
> >     /**
> >
> >      * @param args
> >
> >      * @throws IOException
> >
> >      */
> >
> >     public static void main(String[] args) throws IOException {
> >
> >         // TODO Auto-generated method stub
> >
> >
> >
> >         HBaseCRUD hbaseCRUD = new HBaseCRUD();
> >
> >
> >
> >         /*hbaseCRUD.createTables("CUSTOMERS", "CUSTOMER_INFO");
> >
> >
> >
> >         hbaseCRUD.populateTableData("CUSTOMERS");*/
> >
> >
> >
> >         hbaseCRUD.scanTable("CUSTOMERS", "CUSTOMER_INFO", "EMAIL");
> >
> >     }
> >
> >
> >
> >     private void createTables(String tableName, String...
> > columnFamilyNames)
> >
> >             throws IOException {
> >
> >         // TODO Auto-generated method stub
> >
> >
> >
> >         HTableDescriptor tableDesc = new HTableDescriptor(tableName);
> >
> >
> >
> >         if (!(columnFamilyNames == null || columnFamilyNames.length ==
> 0))
> > {
> >
> >
> >
> >             for (String columnFamilyName : columnFamilyNames) {
> >
> >
> >
> >                 HColumnDescriptor columnFamily = new HColumnDescriptor(
> >
> >                         columnFamilyName);
> >
> >
> >
> >                 tableDesc.addFamily(columnFamily);
> >
> >             }
> >
> >         }
> >
> >
> >
> >         hbaseAdmin.createTable(tableDesc);
> >
> >     }
> >
> >
> >
> >     private void populateTableData(String tableName) throws IOException {
> >
> >         HTableInterface tbl =
> > hTablePool.getTable(Bytes.toBytes(tableName));
> >
> >
> >
> >         List<Put> tblRows = getTableData(tableName);
> >
> >
> >
> >         tbl.close();
> >
> >     }
> >
> >
> >
> >     private List<Put> getTableData(String tableName) {
> >
> >         // TODO Auto-generated method stub
> >
> >
> >
> >         if (tableName == null || tableName.isEmpty())
> >
> >             return null;
> >
> >
> >
> >         /* Pull data from wherever located */
> >
> >
> >
> >         if (tableName.equalsIgnoreCase("CUSTOMERS")) {
> >
> >
> >
> >             /*
> >
> >              * Put p1 = new Put(); p1.add(Bytes.toBytes("CUSTOMER_INFO"),
> >
> >              * Bytes.toBytes("NAME"), value)
> >
> >              */
> >
> >         }
> >
> >
> >
> >         return null;
> >
> >     }
> >
> >
> >
> >     private void scanTable(String tableName, String columnFamilyName,
> >
> >             String... columnNames) throws IOException {
> >
> >
> >
> >         System.out.println("In HBaseCRUD.scanTable(...)");
> >
> >
> >
> >         Scan scan = new Scan();
> >
> >         if (!(columnNames == null || columnNames.length <= 0)) {
> >
> >
> >
> >             for (String columnName : columnNames) {
> >
> >
> >
> >                 scan.addColumn(Bytes.toBytes(columnFamilyName),
> >
> >                         Bytes.toBytes(columnName));
> >
> >             }
> >
> >
> >
> >             Filter filter = new
> ValueFilter(CompareFilter.CompareOp.EQUAL,
> >
> >                     new RegexStringComparator("lntinfotech"));
> >
> >             scan.setFilter(filter);
> >
> >         }
> >
> >
> >
> >         HTableInterface tbl =
> > hTablePool.getTable(Bytes.toBytes(tableName));
> >
> >         ResultScanner scanResults = tbl.getScanner(scan);
> >
> >
> >
> >         for (Result result : scanResults) {
> >
> >             System.out.println("The result is " + result);
> >
> >         }
> >
> >
> >
> >         tbl.close();
> >
> >     }
> >
> > }
> >
> >
> >
> > 7.       The exception I get is :
> >
> >
> > In HBaseCRUD.scanTable(...)
> >
> > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> >
> > SLF4J: Defaulting to no-operation (NOP) logger implementation
> >
> > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> further
> > details.
> >
> > Apr 10, 2013 4:24:54 PM
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper <init>
> >
> > INFO: The identifier of this process is 3648@INFVA03351
> >
> > Apr 10, 2013 4:24:56 PM
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow
> >
> > WARNING: Possibly transient ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
> >
> > Apr 10, 2013 4:24:56 PM org.apache.hadoop.hbase.util.RetryCounter
> > sleepUntilNextRetry
> >
> > INFO: Sleeping 2000ms before retry #1...
> >
> > Apr 10, 2013 4:24:58 PM
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow
> >
> > WARNING: Possibly transient ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
> >
> > Apr 10, 2013 4:24:58 PM org.apache.hadoop.hbase.util.RetryCounter
> > sleepUntilNextRetry
> >
> > INFO: Sleeping 4000ms before retry #2...
> >
> >
> > I have also made the master's entry in my local machine's host file. What
> > can be the error?
> >
> > Thanks and regards !
> >
> >
> > ________________________________
> > The contents of this e-mail and any attachment(s) may contain
> confidential
> > or privileged information for the intended recipient(s). Unintended
> > recipients are prohibited from taking action on the basis of information
> in
> > this e-mail and using or disseminating the information, and must notify
> the
> > sender and delete it from their system. L&T Infotech will not accept
> > responsibility or liability for the accuracy or completeness of, or the
> > presence of any virus or disabling code in this e-mail"
> >
>

RE: Unable to connect from windows desktop to HBase

Posted by Omkar Joshi <Om...@lntinfotech.com>.
Hi Azuryy,

Thanks for the reply !

It solved the issue but I'm not clear as to why this is required; also had to include the Google's protobuf-java-2.4.1.jar whose purpose is unclear to me.

Regards,
Omkar Joshi


-----Original Message-----
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: Monday, April 15, 2013 3:22 PM
To: user@hbase.apache.org
Subject: Re: Unable to connect from windows desktop to HBase

you should put hbase-site.xml under your class path in your java client.

then, don't need to specify zookeeper quorum in your code.


On Mon, Apr 15, 2013 at 5:35 PM, Omkar Joshi <Om...@lntinfotech.com>wrote:

> Attaching the dumps from the HBase UI :
>
> Debug dump :
>
> 55 active threads
> Thread 10536 (IPC Client (47) connection to cldx-1139-1033/
> 172.25.6.69:9000 from hduser):
>   State: TIMED_WAITING
>   Blocked count: 1
>   Waited count: 2
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:706)
>     org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)
> Thread 10535 (IPC Client (47) connection to cldx-1139-1033/
> 172.25.6.69:60000 from hduser):
>   State: TIMED_WAITING
>   Blocked count: 3
>   Waited count: 4
>   Stack:
>     java.lang.Object.wait(Native Method)
>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:534)
>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:579)
> Thread 10121 (1840929932@qtp-1863048490-7):
>   State: RUNNABLE
>   Blocked count: 77
>   Waited count: 415
>   Stack:
>     sun.management.ThreadImpl.getThreadInfo1(Native Method)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
>
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
>
> org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:203)
>
> org.apache.hadoop.http.HttpServer$StackServlet.doGet(HttpServer.java:721)
>     javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     org.mortbay.jetty.Server.handle(Server.java:326)
> Thread 1689 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-3):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 4
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 260 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-2):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 4
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 251 (MASTER_TABLE_OPERATIONS-cldx-1139-1033,60000,1365517046931-0):
>   State: WAITING
>   Blocked count: 25
>   Waited count: 31
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fce66ba
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 82 (cldx-1139-1033,60000,1365517046931-CatalogJanitor):
>   State: TIMED_WAITING
>   Blocked count: 5247
>   Waited count: 6944
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 81 (cldx-1139-1033,60000,1365517046931-BalancerChore):
>   State: TIMED_WAITING
>   Blocked count: 39
>   Waited count: 1736
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 80 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-1):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 3
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 79 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-0):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 3
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 77 (master-cldx-1139-1033,60000,1365517046931-EventThread):
>   State: WAITING
>   Blocked count: 1
>   Waited count: 5
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@10590940
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
> Thread 76
> (master-cldx-1139-1033,60000,1365517046931-SendThread(cldx-1140-1034:2181)):
>   State: RUNNABLE
>   Blocked count: 40
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
>     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> Thread 68 (cldx-1139-1033,60000,1365517046931.timeoutMonitor):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 52060
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 75 (Timer-0):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 17354
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:552)
>     java.util.TimerThread.run(Timer.java:505)
> Thread 74 (307539625@qtp-1863048490-1 - Acceptor0
> SelectChannelConnector@0.0.0.0:60010):
>   State: RUNNABLE
>   Blocked count: 92
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>     org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 71 (master-cldx-1139-1033,60000,1365517046931.archivedHFileCleaner):
>   State: TIMED_WAITING
>   Blocked count: 8718
>   Waited count: 17353
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 70 (master-cldx-1139-1033,60000,1365517046931.oldLogCleaner):
>   State: TIMED_WAITING
>   Blocked count: 8934
>   Waited count: 17353
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 69 (cldx-1139-1033,60000,1365517046931.timerUpdater):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 52061
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 45 (LeaseChecker):
>   State: TIMED_WAITING
>   Blocked count: 1
>   Waited count: 520519
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)
>     java.lang.Thread.run(Thread.java:722)
> Thread 43
> (cldx-1139-1033,60000,1365517046931.splitLogManagerTimeoutMonitor):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 520520
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 9 (master-cldx-1139-1033,60000,1365517046931):
>   State: TIMED_WAITING
>   Blocked count: 322
>   Waited count: 520958
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:55)
>     org.apache.hadoop.hbase.master.HMaster.loop(HMaster.java:489)
>     org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:387)
>     java.lang.Thread.run(Thread.java:722)
> Thread 40 (Timer thread for monitoring jvm):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 52062
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:552)
>     java.util.TimerThread.run(Timer.java:505)
> Thread 39 (Timer thread for monitoring hbase):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 52062
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:552)
>     java.util.TimerThread.run(Timer.java:505)
> Thread 38 (REPL IPC Server handler 2 on 60000):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 37 (REPL IPC Server handler 1 on 60000):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 36 (REPL IPC Server handler 0 on 60000):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 35 (IPC Server handler 9 on 60000):
>   State: WAITING
>   Blocked count: 33
>   Waited count: 17376
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 34 (IPC Server handler 8 on 60000):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 17343
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 33 (IPC Server handler 7 on 60000):
>   State: WAITING
>   Blocked count: 50
>   Waited count: 17393
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 32 (IPC Server handler 6 on 60000):
>   State: WAITING
>   Blocked count: 13
>   Waited count: 17355
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 31 (IPC Server handler 5 on 60000):
>   State: WAITING
>   Blocked count: 27
>   Waited count: 17368
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 30 (IPC Server handler 4 on 60000):
>   State: WAITING
>   Blocked count: 26
>   Waited count: 17370
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 29 (IPC Server handler 3 on 60000):
>   State: WAITING
>   Blocked count: 13
>   Waited count: 17357
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 28 (IPC Server handler 2 on 60000):
>   State: WAITING
>   Blocked count: 57
>   Waited count: 17401
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 27 (IPC Server handler 1 on 60000):
>   State: WAITING
>   Blocked count: 12
>   Waited count: 17356
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 26 (IPC Server handler 0 on 60000):
>   State: WAITING
>   Blocked count: 13
>   Waited count: 17357
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 10 (IPC Server listener on 60000):
>   State: RUNNABLE
>   Blocked count: 17
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:636)
> Thread 23 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRunLoop(HBaseServer.java:825)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:808)
> Thread 25 (main-EventThread):
>   State: WAITING
>   Blocked count: 30
>   Waited count: 55
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f38d8fa
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
> Thread 24 (main-SendThread(cldx-1140-1034:2181)):
>   State: RUNNABLE
>   Blocked count: 50
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
>     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> Thread 21 (Timer thread for monitoring rpc):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 52062
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:552)
>     java.util.TimerThread.run(Timer.java:505)
> Thread 20 (IPC Reader 9 on port 60000):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 19 (IPC Reader 8 on port 60000):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 18 (IPC Reader 7 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 17 (IPC Reader 6 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 16 (IPC Reader 5 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 15 (IPC Reader 4 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 14 (IPC Reader 3 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 13 (IPC Reader 2 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 12 (IPC Reader 1 on port 60000):
>   State: RUNNABLE
>   Blocked count: 3
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 11 (IPC Reader 0 on port 60000):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 5 (Signal Dispatcher):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
> Thread 3 (Finalizer):
>   State: WAITING
>   Blocked count: 539
>   Waited count: 540
>   Waiting on java.lang.ref.ReferenceQueue$Lock@11499ff
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>     java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
> Thread 2 (Reference Handler):
>   State: WAITING
>   Blocked count: 538
>   Waited count: 539
>   Waiting on java.lang.ref.Reference$Lock@6f8a34e6
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.Object.wait(Object.java:503)
>     java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> Thread 1 (main):
>   State: WAITING
>   Blocked count: 16
>   Waited count: 17
>   Waiting on java.lang.Thread@3a246847
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.Thread.join(Thread.java:1258)
>     java.lang.Thread.join(Thread.java:1332)
>     org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:86)
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:158)
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
>     org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
>     org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1944)
>
>
>
>
> Regards,
> Omkar Joshi
>
> -----Original Message-----
> From: Omkar Joshi [mailto:Omkar.Joshi@lntinfotech.com]
> Sent: Monday, April 15, 2013 9:12 AM
> To: user@hbase.apache.org
> Subject: Unable to connect from windows desktop to HBase
>
> Hi,
>
> I'm trying to connect from a Java client running on a Windows desktop
> machine to a remote HBase cluster running in the distributed mode(atop
> 2-node Hadoop cluster).
>
> 1.      On the master(namenode)node :
> hduser@cldx-1139-1033:~/hadoop_ecosystem/apache_hbase/hbase_installation/hbase-0.94.6.1/bin$
> jps
> 9161 HMaster
> 4536 SecondaryNameNode
> 4368 DataNode
> 4645 JobTracker
> 8395 Jps
> 4813 TaskTracker
> 4179 NameNode
> 7717 Main
>
>
>
> 2.       On the slave(datanode)node :
>
>
> hduser@cldx-1140-1034:~$ jps
>
> 5677 HRegionServer
>
> 5559 HQuorumPeer
>
> 2634 TaskTracker
>
> 3260 Jps
>
> 2496 DataNode
>
>
>
> 3.       I also connected to the shell and created a hbase table and also
> able to scan it :
>
>
> hbase(main):004:0> scan 'CUSTOMERS'
>
> ROW                              COLUMN+CELL
>
>  CUSTID12345                     column=CUSTOMER_INFO:EMAIL,
> timestamp=1365600369284, value=Omkar.Joshi@lntinfotech.com
>
>  CUSTID12345                     column=CUSTOMER_INFO:NAME,
> timestamp=1365600052104, value=Omkar Joshi
>
>  CUSTID614                       column=CUSTOMER_INFO:NAME,
> timestamp=1365601350972, value=Prachi Shah
>
> 2 row(s) in 0.8760 seconds
>
>
>
> 4.       The hbase-site.xml has the following configurations:
>
>
> <configuration>
>
> <property>
>
>     <name>hbase.zookeeper.quorum</name>
>
>     <value>cldx-1140-1034</value>
>
>     <description>The directory shared by RegionServers.
>
>     </description>
>
> </property>
>
> <property>
>
>     <name>hbase.zookeeper.property.dataDir</name>
>
>     <value>/home/hduser/hadoop_ecosystem/apache_hbase/zk_datadir</value>
>
>     <description>Property from ZooKeeper's config zoo.cfg.
>
>     The directory where the snapshot is stored.
>
>     </description>
>
> </property>
>
> <property>
>
>     <name>hbase.rootdir</name>
>
>     <value>hdfs://cldx-1139-1033:9000/hbase</value>
>
>     <description>The directory shared by RegionServers.</description>
>
> </property>
>
> <property>
>
>     <name>hbase.cluster.distributed</name>
>
>     <value>true</value>
>
>     <description>The mode the cluster will be in. Possible values are
> false: standalone and pseudo-distributed setups with managed Zookeeper
> true: fully-distributed with unmanaged Zookeeper Quorum (see
> hbase-env.sh)</description>
>
> </property>
>
> </configuration>
>
>
>
> 5.       The Hadoop's core-site.xml has the following settings :
>
>
> <configuration>
>
> <property>
>
> <name>fs.default.name</name>
>
> <value>hdfs://cldx-1139-1033:9000</value>
>
> </property>
>
> </configuration>
>
>
>
> 6.       My java client class is :
>
>
> package client.hbase;
>
>
>
> import java.io.IOException;
>
> import java.util.List;
>
>
>
> import org.apache.hadoop.conf.Configuration;
>
> import org.apache.hadoop.hbase.HBaseConfiguration;
>
> import org.apache.hadoop.hbase.HColumnDescriptor;
>
> import org.apache.hadoop.hbase.HTableDescriptor;
>
> import org.apache.hadoop.hbase.client.HBaseAdmin;
>
> import org.apache.hadoop.hbase.client.HTableInterface;
>
> import org.apache.hadoop.hbase.client.HTablePool;
>
> import org.apache.hadoop.hbase.client.Put;
>
> import org.apache.hadoop.hbase.client.Result;
>
> import org.apache.hadoop.hbase.client.ResultScanner;
>
> import org.apache.hadoop.hbase.client.Scan;
>
> import org.apache.hadoop.hbase.filter.CompareFilter;
>
> import org.apache.hadoop.hbase.filter.Filter;
>
> import org.apache.hadoop.hbase.filter.RegexStringComparator;
>
> import org.apache.hadoop.hbase.filter.ValueFilter;
>
> import org.apache.hadoop.hbase.util.Bytes;
>
>
>
> public class HBaseCRUD {
>
>
>
>     private static Configuration config;
>
>     private static HBaseAdmin hbaseAdmin;
>
>
>
>     private static HTablePool hTablePool;
>
>
>
>     static {
>
>         config = HBaseConfiguration.create();
>
>         config.set("hbase.zookeeper.quorum", "172.25.6.69");
>
>
>
>         hTablePool = new HTablePool(config, 2);
>
>     }
>
>
>
>     /**
>
>      * @param args
>
>      * @throws IOException
>
>      */
>
>     public static void main(String[] args) throws IOException {
>
>         // TODO Auto-generated method stub
>
>
>
>         HBaseCRUD hbaseCRUD = new HBaseCRUD();
>
>
>
>         /*hbaseCRUD.createTables("CUSTOMERS", "CUSTOMER_INFO");
>
>
>
>         hbaseCRUD.populateTableData("CUSTOMERS");*/
>
>
>
>         hbaseCRUD.scanTable("CUSTOMERS", "CUSTOMER_INFO", "EMAIL");
>
>     }
>
>
>
>     private void createTables(String tableName, String...
> columnFamilyNames)
>
>             throws IOException {
>
>         // TODO Auto-generated method stub
>
>
>
>         HTableDescriptor tableDesc = new HTableDescriptor(tableName);
>
>
>
>         if (!(columnFamilyNames == null || columnFamilyNames.length == 0))
> {
>
>
>
>             for (String columnFamilyName : columnFamilyNames) {
>
>
>
>                 HColumnDescriptor columnFamily = new HColumnDescriptor(
>
>                         columnFamilyName);
>
>
>
>                 tableDesc.addFamily(columnFamily);
>
>             }
>
>         }
>
>
>
>         hbaseAdmin.createTable(tableDesc);
>
>     }
>
>
>
>     private void populateTableData(String tableName) throws IOException {
>
>         HTableInterface tbl =
> hTablePool.getTable(Bytes.toBytes(tableName));
>
>
>
>         List<Put> tblRows = getTableData(tableName);
>
>
>
>         tbl.close();
>
>     }
>
>
>
>     private List<Put> getTableData(String tableName) {
>
>         // TODO Auto-generated method stub
>
>
>
>         if (tableName == null || tableName.isEmpty())
>
>             return null;
>
>
>
>         /* Pull data from wherever located */
>
>
>
>         if (tableName.equalsIgnoreCase("CUSTOMERS")) {
>
>
>
>             /*
>
>              * Put p1 = new Put(); p1.add(Bytes.toBytes("CUSTOMER_INFO"),
>
>              * Bytes.toBytes("NAME"), value)
>
>              */
>
>         }
>
>
>
>         return null;
>
>     }
>
>
>
>     private void scanTable(String tableName, String columnFamilyName,
>
>             String... columnNames) throws IOException {
>
>
>
>         System.out.println("In HBaseCRUD.scanTable(...)");
>
>
>
>         Scan scan = new Scan();
>
>         if (!(columnNames == null || columnNames.length <= 0)) {
>
>
>
>             for (String columnName : columnNames) {
>
>
>
>                 scan.addColumn(Bytes.toBytes(columnFamilyName),
>
>                         Bytes.toBytes(columnName));
>
>             }
>
>
>
>             Filter filter = new ValueFilter(CompareFilter.CompareOp.EQUAL,
>
>                     new RegexStringComparator("lntinfotech"));
>
>             scan.setFilter(filter);
>
>         }
>
>
>
>         HTableInterface tbl =
> hTablePool.getTable(Bytes.toBytes(tableName));
>
>         ResultScanner scanResults = tbl.getScanner(scan);
>
>
>
>         for (Result result : scanResults) {
>
>             System.out.println("The result is " + result);
>
>         }
>
>
>
>         tbl.close();
>
>     }
>
> }
>
>
>
> 7.       The exception I get is :
>
>
> In HBaseCRUD.scanTable(...)
>
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>
> SLF4J: Defaulting to no-operation (NOP) logger implementation
>
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
>
> Apr 10, 2013 4:24:54 PM
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper <init>
>
> INFO: The identifier of this process is 3648@INFVA03351
>
> Apr 10, 2013 4:24:56 PM
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow
>
> WARNING: Possibly transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
>
> Apr 10, 2013 4:24:56 PM org.apache.hadoop.hbase.util.RetryCounter
> sleepUntilNextRetry
>
> INFO: Sleeping 2000ms before retry #1...
>
> Apr 10, 2013 4:24:58 PM
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow
>
> WARNING: Possibly transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
>
> Apr 10, 2013 4:24:58 PM org.apache.hadoop.hbase.util.RetryCounter
> sleepUntilNextRetry
>
> INFO: Sleeping 4000ms before retry #2...
>
>
> I have also made the master's entry in my local machine's host file. What
> can be the error?
>
> Thanks and regards !
>
>
> ________________________________
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>

Re: Unable to connect from windows desktop to HBase

Posted by Azuryy Yu <az...@gmail.com>.
you should put hbase-site.xml under your class path in your java client.

then, don't need to specify zookeeper quorum in your code.


On Mon, Apr 15, 2013 at 5:35 PM, Omkar Joshi <Om...@lntinfotech.com>wrote:

> Attaching the dumps from the HBase UI :
>
> Debug dump :
>
> 55 active threads
> Thread 10536 (IPC Client (47) connection to cldx-1139-1033/
> 172.25.6.69:9000 from hduser):
>   State: TIMED_WAITING
>   Blocked count: 1
>   Waited count: 2
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:706)
>     org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)
> Thread 10535 (IPC Client (47) connection to cldx-1139-1033/
> 172.25.6.69:60000 from hduser):
>   State: TIMED_WAITING
>   Blocked count: 3
>   Waited count: 4
>   Stack:
>     java.lang.Object.wait(Native Method)
>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:534)
>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:579)
> Thread 10121 (1840929932@qtp-1863048490-7):
>   State: RUNNABLE
>   Blocked count: 77
>   Waited count: 415
>   Stack:
>     sun.management.ThreadImpl.getThreadInfo1(Native Method)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
>
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
>
> org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:203)
>
> org.apache.hadoop.http.HttpServer$StackServlet.doGet(HttpServer.java:721)
>     javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     org.mortbay.jetty.Server.handle(Server.java:326)
> Thread 1689 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-3):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 4
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 260 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-2):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 4
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 251 (MASTER_TABLE_OPERATIONS-cldx-1139-1033,60000,1365517046931-0):
>   State: WAITING
>   Blocked count: 25
>   Waited count: 31
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fce66ba
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 82 (cldx-1139-1033,60000,1365517046931-CatalogJanitor):
>   State: TIMED_WAITING
>   Blocked count: 5247
>   Waited count: 6944
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 81 (cldx-1139-1033,60000,1365517046931-BalancerChore):
>   State: TIMED_WAITING
>   Blocked count: 39
>   Waited count: 1736
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 80 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-1):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 3
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 79 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-0):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 3
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 77 (master-cldx-1139-1033,60000,1365517046931-EventThread):
>   State: WAITING
>   Blocked count: 1
>   Waited count: 5
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@10590940
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
> Thread 76
> (master-cldx-1139-1033,60000,1365517046931-SendThread(cldx-1140-1034:2181)):
>   State: RUNNABLE
>   Blocked count: 40
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
>     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> Thread 68 (cldx-1139-1033,60000,1365517046931.timeoutMonitor):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 52060
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 75 (Timer-0):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 17354
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:552)
>     java.util.TimerThread.run(Timer.java:505)
> Thread 74 (307539625@qtp-1863048490-1 - Acceptor0
> SelectChannelConnector@0.0.0.0:60010):
>   State: RUNNABLE
>   Blocked count: 92
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>     org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 71 (master-cldx-1139-1033,60000,1365517046931.archivedHFileCleaner):
>   State: TIMED_WAITING
>   Blocked count: 8718
>   Waited count: 17353
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 70 (master-cldx-1139-1033,60000,1365517046931.oldLogCleaner):
>   State: TIMED_WAITING
>   Blocked count: 8934
>   Waited count: 17353
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 69 (cldx-1139-1033,60000,1365517046931.timerUpdater):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 52061
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 45 (LeaseChecker):
>   State: TIMED_WAITING
>   Blocked count: 1
>   Waited count: 520519
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)
>     java.lang.Thread.run(Thread.java:722)
> Thread 43
> (cldx-1139-1033,60000,1365517046931.splitLogManagerTimeoutMonitor):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 520520
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.Chore.run(Chore.java:75)
>     java.lang.Thread.run(Thread.java:722)
> Thread 9 (master-cldx-1139-1033,60000,1365517046931):
>   State: TIMED_WAITING
>   Blocked count: 322
>   Waited count: 520958
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>     org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:55)
>     org.apache.hadoop.hbase.master.HMaster.loop(HMaster.java:489)
>     org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:387)
>     java.lang.Thread.run(Thread.java:722)
> Thread 40 (Timer thread for monitoring jvm):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 52062
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:552)
>     java.util.TimerThread.run(Timer.java:505)
> Thread 39 (Timer thread for monitoring hbase):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 52062
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:552)
>     java.util.TimerThread.run(Timer.java:505)
> Thread 38 (REPL IPC Server handler 2 on 60000):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 37 (REPL IPC Server handler 1 on 60000):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 36 (REPL IPC Server handler 0 on 60000):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 35 (IPC Server handler 9 on 60000):
>   State: WAITING
>   Blocked count: 33
>   Waited count: 17376
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 34 (IPC Server handler 8 on 60000):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 17343
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 33 (IPC Server handler 7 on 60000):
>   State: WAITING
>   Blocked count: 50
>   Waited count: 17393
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 32 (IPC Server handler 6 on 60000):
>   State: WAITING
>   Blocked count: 13
>   Waited count: 17355
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 31 (IPC Server handler 5 on 60000):
>   State: WAITING
>   Blocked count: 27
>   Waited count: 17368
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 30 (IPC Server handler 4 on 60000):
>   State: WAITING
>   Blocked count: 26
>   Waited count: 17370
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 29 (IPC Server handler 3 on 60000):
>   State: WAITING
>   Blocked count: 13
>   Waited count: 17357
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 28 (IPC Server handler 2 on 60000):
>   State: WAITING
>   Blocked count: 57
>   Waited count: 17401
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 27 (IPC Server handler 1 on 60000):
>   State: WAITING
>   Blocked count: 12
>   Waited count: 17356
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 26 (IPC Server handler 0 on 60000):
>   State: WAITING
>   Blocked count: 13
>   Waited count: 17357
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
> Thread 10 (IPC Server listener on 60000):
>   State: RUNNABLE
>   Blocked count: 17
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:636)
> Thread 23 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRunLoop(HBaseServer.java:825)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:808)
> Thread 25 (main-EventThread):
>   State: WAITING
>   Blocked count: 30
>   Waited count: 55
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f38d8fa
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
> Thread 24 (main-SendThread(cldx-1140-1034:2181)):
>   State: RUNNABLE
>   Blocked count: 50
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
>     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> Thread 21 (Timer thread for monitoring rpc):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 52062
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:552)
>     java.util.TimerThread.run(Timer.java:505)
> Thread 20 (IPC Reader 9 on port 60000):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 19 (IPC Reader 8 on port 60000):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 18 (IPC Reader 7 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 17 (IPC Reader 6 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 16 (IPC Reader 5 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 15 (IPC Reader 4 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 14 (IPC Reader 3 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 13 (IPC Reader 2 on port 60000):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 12 (IPC Reader 1 on port 60000):
>   State: RUNNABLE
>   Blocked count: 3
>   Waited count: 2
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 11 (IPC Reader 0 on port 60000):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     java.lang.Thread.run(Thread.java:722)
> Thread 5 (Signal Dispatcher):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
> Thread 3 (Finalizer):
>   State: WAITING
>   Blocked count: 539
>   Waited count: 540
>   Waiting on java.lang.ref.ReferenceQueue$Lock@11499ff
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>     java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
> Thread 2 (Reference Handler):
>   State: WAITING
>   Blocked count: 538
>   Waited count: 539
>   Waiting on java.lang.ref.Reference$Lock@6f8a34e6
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.Object.wait(Object.java:503)
>     java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> Thread 1 (main):
>   State: WAITING
>   Blocked count: 16
>   Waited count: 17
>   Waiting on java.lang.Thread@3a246847
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.Thread.join(Thread.java:1258)
>     java.lang.Thread.join(Thread.java:1332)
>     org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:86)
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:158)
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
>     org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
>     org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1944)
>
>
>
>
> Regards,
> Omkar Joshi
> Technology Office
> Phone : 022-67954364
>
> -----Original Message-----
> From: Omkar Joshi [mailto:Omkar.Joshi@lntinfotech.com]
> Sent: Monday, April 15, 2013 9:12 AM
> To: user@hbase.apache.org
> Subject: Unable to connect from windows desktop to HBase
>
> Hi,
>
> I'm trying to connect from a Java client running on a Windows desktop
> machine to a remote HBase cluster running in the distributed mode(atop
> 2-node Hadoop cluster).
>
> 1.      On the master(namenode)node :
> hduser@cldx-1139-1033:~/hadoop_ecosystem/apache_hbase/hbase_installation/hbase-0.94.6.1/bin$
> jps
> 9161 HMaster
> 4536 SecondaryNameNode
> 4368 DataNode
> 4645 JobTracker
> 8395 Jps
> 4813 TaskTracker
> 4179 NameNode
> 7717 Main
>
>
>
> 2.       On the slave(datanode)node :
>
>
> hduser@cldx-1140-1034:~$ jps
>
> 5677 HRegionServer
>
> 5559 HQuorumPeer
>
> 2634 TaskTracker
>
> 3260 Jps
>
> 2496 DataNode
>
>
>
> 3.       I also connected to the shell and created a hbase table and also
> able to scan it :
>
>
> hbase(main):004:0> scan 'CUSTOMERS'
>
> ROW                              COLUMN+CELL
>
>  CUSTID12345                     column=CUSTOMER_INFO:EMAIL,
> timestamp=1365600369284, value=Omkar.Joshi@lntinfotech.com
>
>  CUSTID12345                     column=CUSTOMER_INFO:NAME,
> timestamp=1365600052104, value=Omkar Joshi
>
>  CUSTID614                       column=CUSTOMER_INFO:NAME,
> timestamp=1365601350972, value=Prachi Shah
>
> 2 row(s) in 0.8760 seconds
>
>
>
> 4.       The hbase-site.xml has the following configurations:
>
>
> <configuration>
>
> <property>
>
>     <name>hbase.zookeeper.quorum</name>
>
>     <value>cldx-1140-1034</value>
>
>     <description>The directory shared by RegionServers.
>
>     </description>
>
> </property>
>
> <property>
>
>     <name>hbase.zookeeper.property.dataDir</name>
>
>     <value>/home/hduser/hadoop_ecosystem/apache_hbase/zk_datadir</value>
>
>     <description>Property from ZooKeeper's config zoo.cfg.
>
>     The directory where the snapshot is stored.
>
>     </description>
>
> </property>
>
> <property>
>
>     <name>hbase.rootdir</name>
>
>     <value>hdfs://cldx-1139-1033:9000/hbase</value>
>
>     <description>The directory shared by RegionServers.</description>
>
> </property>
>
> <property>
>
>     <name>hbase.cluster.distributed</name>
>
>     <value>true</value>
>
>     <description>The mode the cluster will be in. Possible values are
> false: standalone and pseudo-distributed setups with managed Zookeeper
> true: fully-distributed with unmanaged Zookeeper Quorum (see
> hbase-env.sh)</description>
>
> </property>
>
> </configuration>
>
>
>
> 5.       The Hadoop's core-site.xml has the following settings :
>
>
> <configuration>
>
> <property>
>
> <name>fs.default.name</name>
>
> <value>hdfs://cldx-1139-1033:9000</value>
>
> </property>
>
> </configuration>
>
>
>
> 6.       My java client class is :
>
>
> package client.hbase;
>
>
>
> import java.io.IOException;
>
> import java.util.List;
>
>
>
> import org.apache.hadoop.conf.Configuration;
>
> import org.apache.hadoop.hbase.HBaseConfiguration;
>
> import org.apache.hadoop.hbase.HColumnDescriptor;
>
> import org.apache.hadoop.hbase.HTableDescriptor;
>
> import org.apache.hadoop.hbase.client.HBaseAdmin;
>
> import org.apache.hadoop.hbase.client.HTableInterface;
>
> import org.apache.hadoop.hbase.client.HTablePool;
>
> import org.apache.hadoop.hbase.client.Put;
>
> import org.apache.hadoop.hbase.client.Result;
>
> import org.apache.hadoop.hbase.client.ResultScanner;
>
> import org.apache.hadoop.hbase.client.Scan;
>
> import org.apache.hadoop.hbase.filter.CompareFilter;
>
> import org.apache.hadoop.hbase.filter.Filter;
>
> import org.apache.hadoop.hbase.filter.RegexStringComparator;
>
> import org.apache.hadoop.hbase.filter.ValueFilter;
>
> import org.apache.hadoop.hbase.util.Bytes;
>
>
>
> public class HBaseCRUD {
>
>
>
>     private static Configuration config;
>
>     private static HBaseAdmin hbaseAdmin;
>
>
>
>     private static HTablePool hTablePool;
>
>
>
>     static {
>
>         config = HBaseConfiguration.create();
>
>         config.set("hbase.zookeeper.quorum", "172.25.6.69");
>
>
>
>         hTablePool = new HTablePool(config, 2);
>
>     }
>
>
>
>     /**
>
>      * @param args
>
>      * @throws IOException
>
>      */
>
>     public static void main(String[] args) throws IOException {
>
>         // TODO Auto-generated method stub
>
>
>
>         HBaseCRUD hbaseCRUD = new HBaseCRUD();
>
>
>
>         /*hbaseCRUD.createTables("CUSTOMERS", "CUSTOMER_INFO");
>
>
>
>         hbaseCRUD.populateTableData("CUSTOMERS");*/
>
>
>
>         hbaseCRUD.scanTable("CUSTOMERS", "CUSTOMER_INFO", "EMAIL");
>
>     }
>
>
>
>     private void createTables(String tableName, String...
> columnFamilyNames)
>
>             throws IOException {
>
>         // TODO Auto-generated method stub
>
>
>
>         HTableDescriptor tableDesc = new HTableDescriptor(tableName);
>
>
>
>         if (!(columnFamilyNames == null || columnFamilyNames.length == 0))
> {
>
>
>
>             for (String columnFamilyName : columnFamilyNames) {
>
>
>
>                 HColumnDescriptor columnFamily = new HColumnDescriptor(
>
>                         columnFamilyName);
>
>
>
>                 tableDesc.addFamily(columnFamily);
>
>             }
>
>         }
>
>
>
>         hbaseAdmin.createTable(tableDesc);
>
>     }
>
>
>
>     private void populateTableData(String tableName) throws IOException {
>
>         HTableInterface tbl =
> hTablePool.getTable(Bytes.toBytes(tableName));
>
>
>
>         List<Put> tblRows = getTableData(tableName);
>
>
>
>         tbl.close();
>
>     }
>
>
>
>     private List<Put> getTableData(String tableName) {
>
>         // TODO Auto-generated method stub
>
>
>
>         if (tableName == null || tableName.isEmpty())
>
>             return null;
>
>
>
>         /* Pull data from wherever located */
>
>
>
>         if (tableName.equalsIgnoreCase("CUSTOMERS")) {
>
>
>
>             /*
>
>              * Put p1 = new Put(); p1.add(Bytes.toBytes("CUSTOMER_INFO"),
>
>              * Bytes.toBytes("NAME"), value)
>
>              */
>
>         }
>
>
>
>         return null;
>
>     }
>
>
>
>     private void scanTable(String tableName, String columnFamilyName,
>
>             String... columnNames) throws IOException {
>
>
>
>         System.out.println("In HBaseCRUD.scanTable(...)");
>
>
>
>         Scan scan = new Scan();
>
>         if (!(columnNames == null || columnNames.length <= 0)) {
>
>
>
>             for (String columnName : columnNames) {
>
>
>
>                 scan.addColumn(Bytes.toBytes(columnFamilyName),
>
>                         Bytes.toBytes(columnName));
>
>             }
>
>
>
>             Filter filter = new ValueFilter(CompareFilter.CompareOp.EQUAL,
>
>                     new RegexStringComparator("lntinfotech"));
>
>             scan.setFilter(filter);
>
>         }
>
>
>
>         HTableInterface tbl =
> hTablePool.getTable(Bytes.toBytes(tableName));
>
>         ResultScanner scanResults = tbl.getScanner(scan);
>
>
>
>         for (Result result : scanResults) {
>
>             System.out.println("The result is " + result);
>
>         }
>
>
>
>         tbl.close();
>
>     }
>
> }
>
>
>
> 7.       The exception I get is :
>
>
> In HBaseCRUD.scanTable(...)
>
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>
> SLF4J: Defaulting to no-operation (NOP) logger implementation
>
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
>
> Apr 10, 2013 4:24:54 PM
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper <init>
>
> INFO: The identifier of this process is 3648@INFVA03351
>
> Apr 10, 2013 4:24:56 PM
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow
>
> WARNING: Possibly transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
>
> Apr 10, 2013 4:24:56 PM org.apache.hadoop.hbase.util.RetryCounter
> sleepUntilNextRetry
>
> INFO: Sleeping 2000ms before retry #1...
>
> Apr 10, 2013 4:24:58 PM
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow
>
> WARNING: Possibly transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
>
> Apr 10, 2013 4:24:58 PM org.apache.hadoop.hbase.util.RetryCounter
> sleepUntilNextRetry
>
> INFO: Sleeping 4000ms before retry #2...
>
>
> I have also made the master's entry in my local machine's host file. What
> can be the error?
>
> Thanks and regards !
>
>
> ________________________________
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>

RE: Unable to connect from windows desktop to HBase

Posted by Omkar Joshi <Om...@lntinfotech.com>.
Attaching the dumps from the HBase UI :

Debug dump :

55 active threads
Thread 10536 (IPC Client (47) connection to cldx-1139-1033/172.25.6.69:9000 from hduser):
  State: TIMED_WAITING
  Blocked count: 1
  Waited count: 2
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:706)
    org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)
Thread 10535 (IPC Client (47) connection to cldx-1139-1033/172.25.6.69:60000 from hduser):
  State: TIMED_WAITING
  Blocked count: 3
  Waited count: 4
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:534)
    org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:579)
Thread 10121 (1840929932@qtp-1863048490-7):
  State: RUNNABLE
  Blocked count: 77
  Waited count: 415
  Stack:
    sun.management.ThreadImpl.getThreadInfo1(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
    org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
    org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:203)
    org.apache.hadoop.http.HttpServer$StackServlet.doGet(HttpServer.java:721)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
    org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
    org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
    org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
    org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
    org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    org.mortbay.jetty.Server.handle(Server.java:326)
Thread 1689 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-3):
  State: WAITING
  Blocked count: 2
  Waited count: 4
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 260 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-2):
  State: WAITING
  Blocked count: 2
  Waited count: 4
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 251 (MASTER_TABLE_OPERATIONS-cldx-1139-1033,60000,1365517046931-0):
  State: WAITING
  Blocked count: 25
  Waited count: 31
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fce66ba
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 82 (cldx-1139-1033,60000,1365517046931-CatalogJanitor):
  State: TIMED_WAITING
  Blocked count: 5247
  Waited count: 6944
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
    org.apache.hadoop.hbase.Chore.run(Chore.java:75)
    java.lang.Thread.run(Thread.java:722)
Thread 81 (cldx-1139-1033,60000,1365517046931-BalancerChore):
  State: TIMED_WAITING
  Blocked count: 39
  Waited count: 1736
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
    org.apache.hadoop.hbase.Chore.run(Chore.java:75)
    java.lang.Thread.run(Thread.java:722)
Thread 80 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-1):
  State: WAITING
  Blocked count: 2
  Waited count: 3
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 79 (MASTER_OPEN_REGION-cldx-1139-1033,60000,1365517046931-0):
  State: WAITING
  Blocked count: 2
  Waited count: 3
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35d87abc
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 77 (master-cldx-1139-1033,60000,1365517046931-EventThread):
  State: WAITING
  Blocked count: 1
  Waited count: 5
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@10590940
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
Thread 76 (master-cldx-1139-1033,60000,1365517046931-SendThread(cldx-1140-1034:2181)):
  State: RUNNABLE
  Blocked count: 40
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
    org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
Thread 68 (cldx-1139-1033,60000,1365517046931.timeoutMonitor):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 52060
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
    org.apache.hadoop.hbase.Chore.run(Chore.java:75)
    java.lang.Thread.run(Thread.java:722)
Thread 75 (Timer-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 17354
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:552)
    java.util.TimerThread.run(Timer.java:505)
Thread 74 (307539625@qtp-1863048490-1 - Acceptor0 SelectChannelConnector@0.0.0.0:60010):
  State: RUNNABLE
  Blocked count: 92
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 71 (master-cldx-1139-1033,60000,1365517046931.archivedHFileCleaner):
  State: TIMED_WAITING
  Blocked count: 8718
  Waited count: 17353
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
    org.apache.hadoop.hbase.Chore.run(Chore.java:75)
    java.lang.Thread.run(Thread.java:722)
Thread 70 (master-cldx-1139-1033,60000,1365517046931.oldLogCleaner):
  State: TIMED_WAITING
  Blocked count: 8934
  Waited count: 17353
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
    org.apache.hadoop.hbase.Chore.run(Chore.java:75)
    java.lang.Thread.run(Thread.java:722)
Thread 69 (cldx-1139-1033,60000,1365517046931.timerUpdater):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 52061
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
    org.apache.hadoop.hbase.Chore.run(Chore.java:75)
    java.lang.Thread.run(Thread.java:722)
Thread 45 (LeaseChecker):
  State: TIMED_WAITING
  Blocked count: 1
  Waited count: 520519
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)
    java.lang.Thread.run(Thread.java:722)
Thread 43 (cldx-1139-1033,60000,1365517046931.splitLogManagerTimeoutMonitor):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 520520
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
    org.apache.hadoop.hbase.Chore.run(Chore.java:75)
    java.lang.Thread.run(Thread.java:722)
Thread 9 (master-cldx-1139-1033,60000,1365517046931):
  State: TIMED_WAITING
  Blocked count: 322
  Waited count: 520958
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:55)
    org.apache.hadoop.hbase.master.HMaster.loop(HMaster.java:489)
    org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:387)
    java.lang.Thread.run(Thread.java:722)
Thread 40 (Timer thread for monitoring jvm):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 52062
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:552)
    java.util.TimerThread.run(Timer.java:505)
Thread 39 (Timer thread for monitoring hbase):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 52062
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:552)
    java.util.TimerThread.run(Timer.java:505)
Thread 38 (REPL IPC Server handler 2 on 60000):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 37 (REPL IPC Server handler 1 on 60000):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 36 (REPL IPC Server handler 0 on 60000):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b34817a
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 35 (IPC Server handler 9 on 60000):
  State: WAITING
  Blocked count: 33
  Waited count: 17376
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 34 (IPC Server handler 8 on 60000):
  State: WAITING
  Blocked count: 0
  Waited count: 17343
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 33 (IPC Server handler 7 on 60000):
  State: WAITING
  Blocked count: 50
  Waited count: 17393
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 32 (IPC Server handler 6 on 60000):
  State: WAITING
  Blocked count: 13
  Waited count: 17355
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 31 (IPC Server handler 5 on 60000):
  State: WAITING
  Blocked count: 27
  Waited count: 17368
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 30 (IPC Server handler 4 on 60000):
  State: WAITING
  Blocked count: 26
  Waited count: 17370
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 29 (IPC Server handler 3 on 60000):
  State: WAITING
  Blocked count: 13
  Waited count: 17357
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 28 (IPC Server handler 2 on 60000):
  State: WAITING
  Blocked count: 57
  Waited count: 17401
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 27 (IPC Server handler 1 on 60000):
  State: WAITING
  Blocked count: 12
  Waited count: 17356
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 26 (IPC Server handler 0 on 60000):
  State: WAITING
  Blocked count: 13
  Waited count: 17357
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24207655
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
Thread 10 (IPC Server listener on 60000):
  State: RUNNABLE
  Blocked count: 17
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:636)
Thread 23 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRunLoop(HBaseServer.java:825)
    org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:808)
Thread 25 (main-EventThread):
  State: WAITING
  Blocked count: 30
  Waited count: 55
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f38d8fa
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
Thread 24 (main-SendThread(cldx-1140-1034:2181)):
  State: RUNNABLE
  Blocked count: 50
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
    org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
Thread 21 (Timer thread for monitoring rpc):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 52062
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:552)
    java.util.TimerThread.run(Timer.java:505)
Thread 20 (IPC Reader 9 on port 60000):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 19 (IPC Reader 8 on port 60000):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 18 (IPC Reader 7 on port 60000):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 2
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 17 (IPC Reader 6 on port 60000):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 2
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 16 (IPC Reader 5 on port 60000):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 2
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 15 (IPC Reader 4 on port 60000):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 2
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 14 (IPC Reader 3 on port 60000):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 2
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 13 (IPC Reader 2 on port 60000):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 2
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 12 (IPC Reader 1 on port 60000):
  State: RUNNABLE
  Blocked count: 3
  Waited count: 2
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 11 (IPC Reader 0 on port 60000):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    java.lang.Thread.run(Thread.java:722)
Thread 5 (Signal Dispatcher):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
Thread 3 (Finalizer):
  State: WAITING
  Blocked count: 539
  Waited count: 540
  Waiting on java.lang.ref.ReferenceQueue$Lock@11499ff
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
Thread 2 (Reference Handler):
  State: WAITING
  Blocked count: 538
  Waited count: 539
  Waiting on java.lang.ref.Reference$Lock@6f8a34e6
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:503)
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
Thread 1 (main):
  State: WAITING
  Blocked count: 16
  Waited count: 17
  Waiting on java.lang.Thread@3a246847
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Thread.join(Thread.java:1258)
    java.lang.Thread.join(Thread.java:1332)
    org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:86)
    org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:158)
    org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
    org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1944)




Regards,
Omkar Joshi
Technology Office
Phone : 022-67954364

-----Original Message-----
From: Omkar Joshi [mailto:Omkar.Joshi@lntinfotech.com]
Sent: Monday, April 15, 2013 9:12 AM
To: user@hbase.apache.org
Subject: Unable to connect from windows desktop to HBase

Hi,

I'm trying to connect from a Java client running on a Windows desktop machine to a remote HBase cluster running in the distributed mode(atop 2-node Hadoop cluster).

1.      On the master(namenode)node :
hduser@cldx-1139-1033:~/hadoop_ecosystem/apache_hbase/hbase_installation/hbase-0.94.6.1/bin$ jps
9161 HMaster
4536 SecondaryNameNode
4368 DataNode
4645 JobTracker
8395 Jps
4813 TaskTracker
4179 NameNode
7717 Main



2.       On the slave(datanode)node :


hduser@cldx-1140-1034:~$ jps

5677 HRegionServer

5559 HQuorumPeer

2634 TaskTracker

3260 Jps

2496 DataNode



3.       I also connected to the shell and created a hbase table and also able to scan it :


hbase(main):004:0> scan 'CUSTOMERS'

ROW                              COLUMN+CELL

 CUSTID12345                     column=CUSTOMER_INFO:EMAIL, timestamp=1365600369284, value=Omkar.Joshi@lntinfotech.com

 CUSTID12345                     column=CUSTOMER_INFO:NAME, timestamp=1365600052104, value=Omkar Joshi

 CUSTID614                       column=CUSTOMER_INFO:NAME, timestamp=1365601350972, value=Prachi Shah

2 row(s) in 0.8760 seconds



4.       The hbase-site.xml has the following configurations:


<configuration>

<property>

    <name>hbase.zookeeper.quorum</name>

    <value>cldx-1140-1034</value>

    <description>The directory shared by RegionServers.

    </description>

</property>

<property>

    <name>hbase.zookeeper.property.dataDir</name>

    <value>/home/hduser/hadoop_ecosystem/apache_hbase/zk_datadir</value>

    <description>Property from ZooKeeper's config zoo.cfg.

    The directory where the snapshot is stored.

    </description>

</property>

<property>

    <name>hbase.rootdir</name>

    <value>hdfs://cldx-1139-1033:9000/hbase</value>

    <description>The directory shared by RegionServers.</description>

</property>

<property>

    <name>hbase.cluster.distributed</name>

    <value>true</value>

    <description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)</description>

</property>

</configuration>



5.       The Hadoop's core-site.xml has the following settings :


<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://cldx-1139-1033:9000</value>

</property>

</configuration>



6.       My java client class is :


package client.hbase;



import java.io.IOException;

import java.util.List;



import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.hbase.HBaseConfiguration;

import org.apache.hadoop.hbase.HColumnDescriptor;

import org.apache.hadoop.hbase.HTableDescriptor;

import org.apache.hadoop.hbase.client.HBaseAdmin;

import org.apache.hadoop.hbase.client.HTableInterface;

import org.apache.hadoop.hbase.client.HTablePool;

import org.apache.hadoop.hbase.client.Put;

import org.apache.hadoop.hbase.client.Result;

import org.apache.hadoop.hbase.client.ResultScanner;

import org.apache.hadoop.hbase.client.Scan;

import org.apache.hadoop.hbase.filter.CompareFilter;

import org.apache.hadoop.hbase.filter.Filter;

import org.apache.hadoop.hbase.filter.RegexStringComparator;

import org.apache.hadoop.hbase.filter.ValueFilter;

import org.apache.hadoop.hbase.util.Bytes;



public class HBaseCRUD {



    private static Configuration config;

    private static HBaseAdmin hbaseAdmin;



    private static HTablePool hTablePool;



    static {

        config = HBaseConfiguration.create();

        config.set("hbase.zookeeper.quorum", "172.25.6.69");



        hTablePool = new HTablePool(config, 2);

    }



    /**

     * @param args

     * @throws IOException

     */

    public static void main(String[] args) throws IOException {

        // TODO Auto-generated method stub



        HBaseCRUD hbaseCRUD = new HBaseCRUD();



        /*hbaseCRUD.createTables("CUSTOMERS", "CUSTOMER_INFO");



        hbaseCRUD.populateTableData("CUSTOMERS");*/



        hbaseCRUD.scanTable("CUSTOMERS", "CUSTOMER_INFO", "EMAIL");

    }



    private void createTables(String tableName, String... columnFamilyNames)

            throws IOException {

        // TODO Auto-generated method stub



        HTableDescriptor tableDesc = new HTableDescriptor(tableName);



        if (!(columnFamilyNames == null || columnFamilyNames.length == 0)) {



            for (String columnFamilyName : columnFamilyNames) {



                HColumnDescriptor columnFamily = new HColumnDescriptor(

                        columnFamilyName);



                tableDesc.addFamily(columnFamily);

            }

        }



        hbaseAdmin.createTable(tableDesc);

    }



    private void populateTableData(String tableName) throws IOException {

        HTableInterface tbl = hTablePool.getTable(Bytes.toBytes(tableName));



        List<Put> tblRows = getTableData(tableName);



        tbl.close();

    }



    private List<Put> getTableData(String tableName) {

        // TODO Auto-generated method stub



        if (tableName == null || tableName.isEmpty())

            return null;



        /* Pull data from wherever located */



        if (tableName.equalsIgnoreCase("CUSTOMERS")) {



            /*

             * Put p1 = new Put(); p1.add(Bytes.toBytes("CUSTOMER_INFO"),

             * Bytes.toBytes("NAME"), value)

             */

        }



        return null;

    }



    private void scanTable(String tableName, String columnFamilyName,

            String... columnNames) throws IOException {



        System.out.println("In HBaseCRUD.scanTable(...)");



        Scan scan = new Scan();

        if (!(columnNames == null || columnNames.length <= 0)) {



            for (String columnName : columnNames) {



                scan.addColumn(Bytes.toBytes(columnFamilyName),

                        Bytes.toBytes(columnName));

            }



            Filter filter = new ValueFilter(CompareFilter.CompareOp.EQUAL,

                    new RegexStringComparator("lntinfotech"));

            scan.setFilter(filter);

        }



        HTableInterface tbl = hTablePool.getTable(Bytes.toBytes(tableName));

        ResultScanner scanResults = tbl.getScanner(scan);



        for (Result result : scanResults) {

            System.out.println("The result is " + result);

        }



        tbl.close();

    }

}



7.       The exception I get is :


In HBaseCRUD.scanTable(...)

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Apr 10, 2013 4:24:54 PM org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper <init>

INFO: The identifier of this process is 3648@INFVA03351

Apr 10, 2013 4:24:56 PM org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow

WARNING: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

Apr 10, 2013 4:24:56 PM org.apache.hadoop.hbase.util.RetryCounter sleepUntilNextRetry

INFO: Sleeping 2000ms before retry #1...

Apr 10, 2013 4:24:58 PM org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper retryOrThrow

WARNING: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

Apr 10, 2013 4:24:58 PM org.apache.hadoop.hbase.util.RetryCounter sleepUntilNextRetry

INFO: Sleeping 4000ms before retry #2...


I have also made the master's entry in my local machine's host file. What can be the error?

Thanks and regards !


________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"