You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vamshi Krishna <va...@gmail.com> on 2013/08/23 13:47:03 UTC
Will hbase automatically distribute the data across region servers or NOT..??
Hi all,
I set up 2 node hbase cluster and i created a table and pumped
10million records. I set the in hbase-site.xml the following property as
shown.
<property>
<name>hbase.hregion.max.filesize</name>
<value>100</value>
</property>
[Over written the default 256MB with 100MB ].
I am sure, the data i inserted into hbase table would be more than 100MB.
In this case will not hbase automatically distribute the multiple regions
resulted in after region splitting, across the region servers..??
In my case, I see all the 215 regions with above property set to 50MB,
residing on the same machine, where I am running master and region server.
Do we need to distribute the data explicitly by any means..?
I guess that should be done automatically by load balancer of Hbase , right?
Please someone help me..!
--
*Regards*
*
Vamshi Krishna
*
Re: Will hbase automatically distribute the data across region
servers or NOT..??
Posted by Samir Ahmic <ah...@gmail.com>.
Hi Vamshi,
What version of hbase you are using ? Regarding *hbase.hregion.max.filesize
*property from hbase-site.xml you should check this from hbase-default.xml :
<property>
<name>hbase.hregion.max.filesize</name>
<value>*1073741824*0</value> ****value is in bytes*
<description>
Maximum HStoreFile size. If any one of a column families' HStoreFiles
has
grown to exceed this value, the hosting HRegion is split in two.
Default: 10G.
</description>
</property>
Your have configured 100 bytes that is to very small value for region size
that is why you ended with 215 regions for such small amount of data.
Regarding region distribution hbase balancer sholud take care that regions
are evenly distributed on servers. If you want to check is balance is
enabled (it is by default) you can go to hbase shell and run
'balance_switch' command.
Cheers:)
On Fri, Aug 23, 2013 at 1:47 PM, Vamshi Krishna <va...@gmail.com>wrote:
> Hi all,
> I set up 2 node hbase cluster and i created a table and pumped
> 10million records. I set the in hbase-site.xml the following property as
> shown.
> <property>
> <name>hbase.hregion.max.filesize</name>
> <value>100</value>
> </property>
> [Over written the default 256MB with 100MB ].
> I am sure, the data i inserted into hbase table would be more than 100MB.
> In this case will not hbase automatically distribute the multiple regions
> resulted in after region splitting, across the region servers..??
>
> In my case, I see all the 215 regions with above property set to 50MB,
> residing on the same machine, where I am running master and region server.
>
> Do we need to distribute the data explicitly by any means..?
> I guess that should be done automatically by load balancer of Hbase ,
> right?
>
> Please someone help me..!
>
> --
> *Regards*
> *
> Vamshi Krishna
> *
>
Re: Will hbase automatically distribute the data across region
servers or NOT..??
Posted by Mike Axiak <mi...@axiak.net>.
Is your balancer enabled?
On Aug 23, 2013 10:22 AM, "Vamshi Krishna" <va...@gmail.com> wrote:
> No that is 10MB itself. Just to observe the region splitting with respect
> to the amount of data i am inserting in to hbase.
> So, here i am inserting 40-50mb data and fixing that property to 10mb and
> checking the region splitting happening.
> But the intersting thing is regions got split BUT they are not being
> distributed across other servers.
> Whatever regions formed from the created tables on machine-1, all of them
> are residing on the same machine-1 not being moved to other machine.
>
>
>
>
> On Fri, Aug 23, 2013 at 7:40 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
> >wrote:
>
> > Vamshi, max value for hbase.hregion.max.filesize to 10MB seems too small.
> > Did you mean 10GB?
> >
> >
> > Regards,
> > Dhaval
> >
> >
> > ________________________________
> > From: Vamshi Krishna <va...@gmail.com>
> > To: user@hbase.apache.org; zhoushuaifeng <zh...@gmail.com>
> > Sent: Friday, 23 August 2013 9:38 AM
> > Subject: Re: Will hbase automatically distribute the data across region
> > servers or NOT..??
> >
> >
> > Thanks for the clarifications.
> > I am using hbase-0.94.10 and zookeepr-3.4.5
> > But I am running into different issues .
> > I set hbase.hregion.max.filesize to 10Mb and i am inserting 10 million
> > rows in to hbase table. During the insertion after some time, suddenly
> > master is going down. I don't know what is the reason for such peculiar
> > behavior.
> > I found in master log below content and not able to make out what exactly
> > the mistake. Please somebody help.
> >
> > master-log:
> >
> > 2013-08-23 18:56:36,865 FATAL org.apache.hadoop.hbase.master.HMaster:
> > Master server abort: loaded coprocessors are: []
> > 2013-08-23 18:56:36,866 FATAL org.apache.hadoop.hbase.master.HMaster:
> > Unexpected state :
> >
> >
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> > state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273
> ..
> > Cannot transit it to OFFLINE.
> > java.lang.IllegalStateException: Unexpected state :
> >
> >
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> > state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273
> ..
> > Cannot transit it to OFFLINE.
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> > at
> >
> >
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> > at
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> > at java.lang.Thread.run(Thread.java:662)
> > 2013-08-23 18:56:36,867 INFO org.apache.hadoop.hbase.master.HMaster:
> > Aborting
> > 2013-08-23 18:56:36,867 DEBUG org.apache.hadoop.hbase.master.HMaster:
> > Stopping service threads
> > 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> > server on 60000
> > 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 0 on 60000: exiting
> > 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 5 on 60000: exiting
> > 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 3 on 60000: exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> > IPC Server listener on 60000
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> > Server handler 2 on 60000: exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> > Server handler 1 on 60000: exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster$2:
> > vamshi,60000,1377263788019-BalancerChore exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster:
> > Stopping infoServer
> > 2013-08-23 18:56:36,873 INFO
> > org.apache.hadoop.hbase.master.cleaner.HFileCleaner:
> > master-vamshi,60000,1377263788019.archivedHFileCleaner exiting
> > 2013-08-23 18:56:36,873 INFO
> org.apache.hadoop.hbase.master.CatalogJanitor:
> > vamshi,60000,1377263788019-CatalogJanitor exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> > Server handler 0 on 60000: exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 9 on 60000: exiting
> > 2013-08-23 18:56:36,874 INFO org.mortbay.log: Stopped
> > SelectChannelConnector@0.0.0.0:60010
> > 2013-08-23 18:56:36,874 INFO
> > org.apache.hadoop.hbase.master.cleaner.LogCleaner:
> > master-vamshi,60000,1377263788019.oldLogCleaner exiting
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 1 on 60000: exiting
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 7 on 60000: exiting
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 6 on 60000: exiting
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 8 on 60000: exiting
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> > IPC Server Responder
> > 2013-08-23 18:56:36,876 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> > IPC Server Responder
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 2 on 60000: exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 4 on 60000: exiting
> > 2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> > master:60000-0x140ab519b0f0000 Unable to set watcher on znode
> > (/hbase/unassigned/05e30711673614f6b41a364c76f3f05f)
> > java.lang.InterruptedException
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:485)
> > at
> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
> > at
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:271)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> > at
> >
> >
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> > at
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> > at java.lang.Thread.run(Thread.java:662)
> > 2013-08-23 18:56:36,876 WARN
> > org.apache.hadoop.hbase.master.AssignmentManager: Attempted to
> create/force
> > node into OFFLINE state before completing assignment but failed to do so
> > for
> >
> >
> scores,\x00\x00\x00\x00\x00\x08b8,1377264147374.39794b7deea3203fc260756f5038d6f8.
> > state=OFFLINE, ts=1377264396802, server=null
> > 2013-08-23 18:56:36,876 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> > master:60000-0x140ab519b0f0000 Unable to get data of znode
> > /hbase/unassigned/d476f8442ce31de90b60080b74daf47f
> > java.lang.InterruptedException
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:485)
> > at
> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
> > at
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:746)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.ZKAssign.getDataNoWatch(ZKAssign.java:904)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:283)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> > at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> > at
> >
> >
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> > at
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> > at java.lang.Thread.run(Thread.java:662)
> > 2013-08-23 18:56:36,877 WARN
> > org.apache.hadoop.hbase.master.AssignmentManager: Attempted to
> create/force
> > node into OFFLINE state before completing assignment but failed to do so
> > for
> >
> >
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> > state=OFFLINE, ts=1377264396862, server=null
> > 2013-08-23 18:56:36,877 WARN
> > org.apache.hadoop.hbase.master.AssignmentManager: Attempted to
> create/force
> > node into OFFLINE state before completing assignment but failed to do so
> > for
> >
> >
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> > state=OFFLINE, ts=1377264396813, server=null
> > 2013-08-23 18:56:36,882 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_FAILED_OPEN,
> server=vamshi_RS,60020,1377263792053,
> > region=d476f8442ce31de90b60080b74daf47f
> > 2013-08-23 18:56:36,882 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
> > for
> >
> >
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> > destination server is vamshi,60020,1377263789273
> > 2013-08-23 18:56:36,882 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
> > plan was found (or we are ignoring an existing plan) for
> >
> >
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> > so generated a random one;
> >
> >
> hri=scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.,
> > src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1)
> available
> > servers
> > 2013-08-23 18:56:36,882 ERROR
> > org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
> > [ClosedRegionHandler-vamshi,60000,1377263788019-38] because the executor
> is
> > missing. Is this process shutting down?
> > 2013-08-23 18:56:36,906 DEBUG
> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
> > org.apache.hadoop.hbase.catalog.CatalogTracker@451415c8
> > 2013-08-23 18:56:36,906 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor:
> > vamshi,60000,1377263788019.timeoutMonitor exiting
> > 2013-08-23 18:56:36,906 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater:
> > vamshi,60000,1377263788019.timerUpdater exiting
> > 2013-08-23 18:56:36,907 INFO
> > org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor:
> > vamshi,60000,1377263788019.splitLogManagerTimeoutMonitor exiting
> > 2013-08-23 18:56:36,910 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_FAILED_OPEN,
> server=vamshi_RS,60020,1377263792053,
> > region=05e30711673614f6b41a364c76f3f05f
> > 2013-08-23 18:56:36,911 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
> > for
> >
> >
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> > destination server is vamshi,60020,1377263789273
> > 2013-08-23 18:56:36,911 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
> > plan was found (or we are ignoring an existing plan) for
> >
> >
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> > so generated a random one;
> >
> >
> hri=scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.,
> > src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1)
> available
> > servers
> > 2013-08-23 18:56:36,911 ERROR
> > org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
> > [ClosedRegionHandler-vamshi,60000,1377263788019-39] because the executor
> is
> > missing. Is this process shutting down?
> > 2013-08-23 18:56:36,912 WARN
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> transient
> > ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for
> > /hbase/unassigned/d476f8442ce31de90b60080b74daf47f
> > 2013-08-23 18:56:36,912 INFO org.apache.hadoop.hbase.util.RetryCounter:
> > Sleeping 2000ms before retry #1...
> > 2013-08-23 18:56:36,914 INFO org.apache.zookeeper.ZooKeeper: Session:
> > 0x140ab519b0f0000 closed
> > 2013-08-23 18:56:36,914 INFO org.apache.hadoop.hbase.master.HMaster:
> > HMaster main thread exiting
> > 2013-08-23 18:56:36,914 ERROR
> > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
> > java.lang.RuntimeException: HMaster Aborted
> > at
> >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
> > at
> >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at
> >
> >
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
> > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2100)
> >
> >
> >
> > My hbase-site.xml :
> >
> > <configuration>
> > <property>
> > <name>hbase.rootdir</name>
> >
> > <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/hbstmp/</value>
> > </property>
> >
> > <property>
> > <name>hbase.cluster.distributed</name>
> > <value>true</value>
> > </property>
> > <property>
> > <name>hbase.master</name>
> > <value>vamshi</value>
> > </property>
> > <property>
> > <name>hbase.zookeeper.property.clientPort</name>
> > <value>2181</value>
> > </property>
> >
> >
> > <property>
> > <name>hbase.hregion.max.filesize</name>
> > <value>10485760</value>
> > </property>
> >
> >
> >
> > <property>
> > <name>hbase.zookeeper.quorum</name>
> > <value>vamshi</value>
> > </property>
> > <property>
> > <name>hbase.zookeeper.property.dataDir</name>
> > <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/zkptmp</value>
> > </property>
> >
> > <property>
> > <name>hbase.zookeeper.property.maxClientCnxns</name>
> > <value>1024</value>
> > </property>
> >
> > <property>
> > <name>hbase.coprocessor.user.region.classes</name>
> > <value>com.bil.coproc.ColumnAggregationEndpoint</value>
> > </property>
> > </configuration>
> >
> >
> >
> >
> > On Fri, Aug 23, 2013 at 7:00 PM, Frank Chow <zh...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > You may should check if the compact is on. If data size in a region is
> > max
> > > than the limition, region will split and balance after a major
> > > compaction(Usually occur automatically).
> > > You can manually by run the compact operaction by the shell commond:
> > > compact <tableName>, or major_compact <tableName>
> > >
> > >
> > >
> > >
> > > Frank Chow
> >
> >
> >
> >
> > --
> > *Regards*
> > *
> > Vamshi Krishna
> > *
> >
>
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *
>
Re: Re: Will hbase automatically distribute the data across region servers or NOT..??
Posted by Frank Chow <zh...@gmail.com>.
Hi,
According the error message in the master log, there maybe some inconsistencies in the configuration, check the configuration on all nodes, if the properity below is configured, and if it's inconsistent.
<property>
<name>hbase.metrics.showTableName</name>
<value>true</value>
<description>Whether to include the prefix "tbl.tablename" in per-column family metrics.
If true, for each metric M, per-cf metrics will be reported for tbl.T.cf.CF.M, if false,
per-cf metrics will be aggregated by column-family across tables, and reported for cf.CF.M.
In both cases, the aggregated metric M across tables and cfs will be reported.
</description>
</property>
Frank Chow
Re: Re: Will hbase automatically distribute the data across region
servers or NOT..??
Posted by Vamshi Krishna <va...@gmail.com>.
Facing a new problem again.
Each time i start hbase and try to create a table on hbase shell, i see
hbase(main):002:0> create 'student', 'marks'
ERROR: org.apache.hadoop.hbase.PleaseHoldException:
org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
I guessed some administrative work is in progress and noticed the
http://<hostname>:60010
(hbase master web UI).
For so much time i observe the below status and not able to stop the hbase
also during this time.
Start Time Description State Status Mon Aug 26 11:35:21 IST 2013 Master
startup RUNNING (since 15mins, 47sec ago) Submit log splitting work of
non-meta region servers (since 15mins, 38sec ago)
Is some thing to do with the below property? I have commented and
uncommented the below property in my hbase-site.xml, still i see the same
thing in both cases.
<property>
<name>hbase.master.distributed.log.splitting</name>
<value>true</value>
</property>
In master log, i don't see any thing wrong, except below line when i
searched with a string 'error'.
2013-08-26 11:35:22,463 ERROR
org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics: Inconsistent
configuration. Previous configuration for using table name in metrics:
true, new configuration: false
2013-08-26 11:35:22,467 DEBUG
org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with
CacheConfig:disabled
.
.
.
(I don't know whether the above line has got some significant effect.)
Could any one help..
On Sun, Aug 25, 2013 at 4:53 PM, Frank Chow <zh...@gmail.com> wrote:
> Hi,
>
> In order to make the children online faster, the policy is to make them
> online on the same regionserver as parent. Then balancer make them
> distributed on other regionsers if overload. It's decided by the parameter
> hbase.regions.slop.
>
> <property>
> <name>hbase.regions.slop</name>
> <value>0.2</value>
> <description>Rebalance if any regionserver has average + (average *
> slop) regions.
> Default is 20% slop.
> </description>
> </property>
>
>
>
>
> Frank Chow
>
> From: Vamshi Krishna
> Date: 2013-08-23 22:51
> To: user; Dhaval Shah
> Subject: Re: Will hbase automatically distribute the data across region
> servers or NOT..??
> No that is 10MB itself. Just to observe the region splitting with respect
> to the amount of data i am inserting in to hbase.
> So, here i am inserting 40-50mb data and fixing that property to 10mb and
> checking the region splitting happening.
> But the intersting thing is regions got split BUT they are not being
> distributed across other servers.
> Whatever regions formed from the created tables on machine-1, all of them
> are residing on the same machine-1 not being moved to other machine.
>
--
*Regards*
*
Vamshi Krishna
*
Re: Re: Will hbase automatically distribute the data across region servers or NOT..??
Posted by Frank Chow <zh...@gmail.com>.
Hi,
In order to make the children online faster, the policy is to make them online on the same regionserver as parent. Then balancer make them distributed on other regionsers if overload. It's decided by the parameter hbase.regions.slop.
<property>
<name>hbase.regions.slop</name>
<value>0.2</value>
<description>Rebalance if any regionserver has average + (average * slop) regions.
Default is 20% slop.
</description>
</property>
Frank Chow
From: Vamshi Krishna
Date: 2013-08-23 22:51
To: user; Dhaval Shah
Subject: Re: Will hbase automatically distribute the data across region servers or NOT..??
No that is 10MB itself. Just to observe the region splitting with respect
to the amount of data i am inserting in to hbase.
So, here i am inserting 40-50mb data and fixing that property to 10mb and
checking the region splitting happening.
But the intersting thing is regions got split BUT they are not being
distributed across other servers.
Whatever regions formed from the created tables on machine-1, all of them
are residing on the same machine-1 not being moved to other machine.
Re: Will hbase automatically distribute the data across region servers or NOT..??
Posted by Dhaval Shah <pr...@yahoo.co.in>.
Ok. The balancer runs as a separate thread (there is a config to set how often the thread wakes up but can't remember off the top of my head). Maybe if you wait long enough, it will balance eventually. Another thing you can try is run the balancer from hbase shell and see what you get back. If you get back a true, it means it should balance. If you get back a false, look at hbase master logs to see whats happening. I once had a scenario where my Unix accounts were messed up (2 users - hbase and another user mapped to the same unix ID and HDFS thought the user did not have the permissions to write to the HBase files on HDFS) and balancer did not run due to this exception.
Another thing is (I think!) balancer generally does not run when regions are splitting. So its possible in your case that your regions are splitting so often (due to 10MB limit) that the balancer cannot be run since your regions are not stationary
Regards,
Dhaval
________________________________
From: Vamshi Krishna <va...@gmail.com>
To: user@hbase.apache.org; Dhaval Shah <pr...@yahoo.co.in>
Sent: Friday, 23 August 2013 10:21 AM
Subject: Re: Will hbase automatically distribute the data across region servers or NOT..??
No that is 10MB itself. Just to observe the region splitting with respect
to the amount of data i am inserting in to hbase.
So, here i am inserting 40-50mb data and fixing that property to 10mb and
checking the region splitting happening.
But the intersting thing is regions got split BUT they are not being
distributed across other servers.
Whatever regions formed from the created tables on machine-1, all of them
are residing on the same machine-1 not being moved to other machine.
On Fri, Aug 23, 2013 at 7:40 PM, Dhaval Shah <pr...@yahoo.co.in>wrote:
> Vamshi, max value for hbase.hregion.max.filesize to 10MB seems too small.
> Did you mean 10GB?
>
>
> Regards,
> Dhaval
>
>
> ________________________________
> From: Vamshi Krishna <va...@gmail.com>
> To: user@hbase.apache.org; zhoushuaifeng <zh...@gmail.com>
> Sent: Friday, 23 August 2013 9:38 AM
> Subject: Re: Will hbase automatically distribute the data across region
> servers or NOT..??
>
>
> Thanks for the clarifications.
> I am using hbase-0.94.10 and zookeepr-3.4.5
> But I am running into different issues .
> I set hbase.hregion.max.filesize to 10Mb and i am inserting 10 million
> rows in to hbase table. During the insertion after some time, suddenly
> master is going down. I don't know what is the reason for such peculiar
> behavior.
> I found in master log below content and not able to make out what exactly
> the mistake. Please somebody help.
>
> master-log:
>
> 2013-08-23 18:56:36,865 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> 2013-08-23 18:56:36,866 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state :
>
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
> Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
>
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
> Cannot transit it to OFFLINE.
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> at
>
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.hbase.master.HMaster:
> Aborting
> 2013-08-23 18:56:36,867 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Stopping service threads
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60000
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60000: exiting
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60000: exiting
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server listener on 60000
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> Server handler 2 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> Server handler 1 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster$2:
> vamshi,60000,1377263788019-BalancerChore exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster:
> Stopping infoServer
> 2013-08-23 18:56:36,873 INFO
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner:
> master-vamshi,60000,1377263788019.archivedHFileCleaner exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
> vamshi,60000,1377263788019-CatalogJanitor exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> Server handler 0 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.mortbay.log: Stopped
> SelectChannelConnector@0.0.0.0:60010
> 2013-08-23 18:56:36,874 INFO
> org.apache.hadoop.hbase.master.cleaner.LogCleaner:
> master-vamshi,60000,1377263788019.oldLogCleaner exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 1 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 6 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 8 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server Responder
> 2013-08-23 18:56:36,876 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server Responder
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60000: exiting
> 2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> master:60000-0x140ab519b0f0000 Unable to set watcher on znode
> (/hbase/unassigned/05e30711673614f6b41a364c76f3f05f)
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:271)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> at
>
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> 2013-08-23 18:56:36,876 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
> node into OFFLINE state before completing assignment but failed to do so
> for
>
> scores,\x00\x00\x00\x00\x00\x08b8,1377264147374.39794b7deea3203fc260756f5038d6f8.
> state=OFFLINE, ts=1377264396802, server=null
> 2013-08-23 18:56:36,876 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> master:60000-0x140ab519b0f0000 Unable to get data of znode
> /hbase/unassigned/d476f8442ce31de90b60080b74daf47f
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149)
> at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:746)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZKAssign.getDataNoWatch(ZKAssign.java:904)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:283)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> at
>
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> 2013-08-23 18:56:36,877 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
> node into OFFLINE state before completing assignment but failed to do so
> for
>
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> state=OFFLINE, ts=1377264396862, server=null
> 2013-08-23 18:56:36,877 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
> node into OFFLINE state before completing assignment but failed to do so
> for
>
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> state=OFFLINE, ts=1377264396813, server=null
> 2013-08-23 18:56:36,882 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
> region=d476f8442ce31de90b60080b74daf47f
> 2013-08-23 18:56:36,882 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
> for
>
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> destination server is vamshi,60020,1377263789273
> 2013-08-23 18:56:36,882 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
> plan was found (or we are ignoring an existing plan) for
>
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> so generated a random one;
>
> hri=scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.,
> src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
> servers
> 2013-08-23 18:56:36,882 ERROR
> org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
> [ClosedRegionHandler-vamshi,60000,1377263788019-38] because the executor is
> missing. Is this process shutting down?
> 2013-08-23 18:56:36,906 DEBUG
> org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
> org.apache.hadoop.hbase.catalog.CatalogTracker@451415c8
> 2013-08-23 18:56:36,906 INFO
> org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor:
> vamshi,60000,1377263788019.timeoutMonitor exiting
> 2013-08-23 18:56:36,906 INFO
> org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater:
> vamshi,60000,1377263788019.timerUpdater exiting
> 2013-08-23 18:56:36,907 INFO
> org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor:
> vamshi,60000,1377263788019.splitLogManagerTimeoutMonitor exiting
> 2013-08-23 18:56:36,910 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
> region=05e30711673614f6b41a364c76f3f05f
> 2013-08-23 18:56:36,911 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
> for
>
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> destination server is vamshi,60020,1377263789273
> 2013-08-23 18:56:36,911 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
> plan was found (or we are ignoring an existing plan) for
>
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> so generated a random one;
>
> hri=scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.,
> src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
> servers
> 2013-08-23 18:56:36,911 ERROR
> org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
> [ClosedRegionHandler-vamshi,60000,1377263788019-39] because the executor is
> missing. Is this process shutting down?
> 2013-08-23 18:56:36,912 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
> ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for
> /hbase/unassigned/d476f8442ce31de90b60080b74daf47f
> 2013-08-23 18:56:36,912 INFO org.apache.hadoop.hbase.util.RetryCounter:
> Sleeping 2000ms before retry #1...
> 2013-08-23 18:56:36,914 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x140ab519b0f0000 closed
> 2013-08-23 18:56:36,914 INFO org.apache.hadoop.hbase.master.HMaster:
> HMaster main thread exiting
> 2013-08-23 18:56:36,914 ERROR
> org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
> java.lang.RuntimeException: HMaster Aborted
> at
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
> at
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2100)
>
>
>
> My hbase-site.xml :
>
> <configuration>
> <property>
> <name>hbase.rootdir</name>
>
> <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/hbstmp/</value>
> </property>
>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.master</name>
> <value>vamshi</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.clientPort</name>
> <value>2181</value>
> </property>
>
>
> <property>
> <name>hbase.hregion.max.filesize</name>
> <value>10485760</value>
> </property>
>
>
>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>vamshi</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/zkptmp</value>
> </property>
>
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>1024</value>
> </property>
>
> <property>
> <name>hbase.coprocessor.user.region.classes</name>
> <value>com.bil.coproc.ColumnAggregationEndpoint</value>
> </property>
> </configuration>
>
>
>
>
> On Fri, Aug 23, 2013 at 7:00 PM, Frank Chow <zh...@gmail.com>
> wrote:
>
> > Hi,
> > You may should check if the compact is on. If data size in a region is
> max
> > than the limition, region will split and balance after a major
> > compaction(Usually occur automatically).
> > You can manually by run the compact operaction by the shell commond:
> > compact <tableName>, or major_compact <tableName>
> >
> >
> >
> >
> > Frank Chow
>
>
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *
>
--
*Regards*
*
Vamshi Krishna
*
Re: Will hbase automatically distribute the data across region
servers or NOT..??
Posted by Vamshi Krishna <va...@gmail.com>.
No that is 10MB itself. Just to observe the region splitting with respect
to the amount of data i am inserting in to hbase.
So, here i am inserting 40-50mb data and fixing that property to 10mb and
checking the region splitting happening.
But the intersting thing is regions got split BUT they are not being
distributed across other servers.
Whatever regions formed from the created tables on machine-1, all of them
are residing on the same machine-1 not being moved to other machine.
On Fri, Aug 23, 2013 at 7:40 PM, Dhaval Shah <pr...@yahoo.co.in>wrote:
> Vamshi, max value for hbase.hregion.max.filesize to 10MB seems too small.
> Did you mean 10GB?
>
>
> Regards,
> Dhaval
>
>
> ________________________________
> From: Vamshi Krishna <va...@gmail.com>
> To: user@hbase.apache.org; zhoushuaifeng <zh...@gmail.com>
> Sent: Friday, 23 August 2013 9:38 AM
> Subject: Re: Will hbase automatically distribute the data across region
> servers or NOT..??
>
>
> Thanks for the clarifications.
> I am using hbase-0.94.10 and zookeepr-3.4.5
> But I am running into different issues .
> I set hbase.hregion.max.filesize to 10Mb and i am inserting 10 million
> rows in to hbase table. During the insertion after some time, suddenly
> master is going down. I don't know what is the reason for such peculiar
> behavior.
> I found in master log below content and not able to make out what exactly
> the mistake. Please somebody help.
>
> master-log:
>
> 2013-08-23 18:56:36,865 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> 2013-08-23 18:56:36,866 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state :
>
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
> Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
>
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
> Cannot transit it to OFFLINE.
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> at
>
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.hbase.master.HMaster:
> Aborting
> 2013-08-23 18:56:36,867 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Stopping service threads
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60000
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60000: exiting
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60000: exiting
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server listener on 60000
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> Server handler 2 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> Server handler 1 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster$2:
> vamshi,60000,1377263788019-BalancerChore exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster:
> Stopping infoServer
> 2013-08-23 18:56:36,873 INFO
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner:
> master-vamshi,60000,1377263788019.archivedHFileCleaner exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
> vamshi,60000,1377263788019-CatalogJanitor exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> Server handler 0 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.mortbay.log: Stopped
> SelectChannelConnector@0.0.0.0:60010
> 2013-08-23 18:56:36,874 INFO
> org.apache.hadoop.hbase.master.cleaner.LogCleaner:
> master-vamshi,60000,1377263788019.oldLogCleaner exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 1 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 6 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 8 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server Responder
> 2013-08-23 18:56:36,876 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server Responder
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60000: exiting
> 2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> master:60000-0x140ab519b0f0000 Unable to set watcher on znode
> (/hbase/unassigned/05e30711673614f6b41a364c76f3f05f)
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:271)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> at
>
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> 2013-08-23 18:56:36,876 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
> node into OFFLINE state before completing assignment but failed to do so
> for
>
> scores,\x00\x00\x00\x00\x00\x08b8,1377264147374.39794b7deea3203fc260756f5038d6f8.
> state=OFFLINE, ts=1377264396802, server=null
> 2013-08-23 18:56:36,876 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> master:60000-0x140ab519b0f0000 Unable to get data of znode
> /hbase/unassigned/d476f8442ce31de90b60080b74daf47f
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149)
> at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:746)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZKAssign.getDataNoWatch(ZKAssign.java:904)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:283)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> at
>
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> 2013-08-23 18:56:36,877 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
> node into OFFLINE state before completing assignment but failed to do so
> for
>
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> state=OFFLINE, ts=1377264396862, server=null
> 2013-08-23 18:56:36,877 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
> node into OFFLINE state before completing assignment but failed to do so
> for
>
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> state=OFFLINE, ts=1377264396813, server=null
> 2013-08-23 18:56:36,882 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
> region=d476f8442ce31de90b60080b74daf47f
> 2013-08-23 18:56:36,882 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
> for
>
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> destination server is vamshi,60020,1377263789273
> 2013-08-23 18:56:36,882 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
> plan was found (or we are ignoring an existing plan) for
>
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> so generated a random one;
>
> hri=scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.,
> src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
> servers
> 2013-08-23 18:56:36,882 ERROR
> org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
> [ClosedRegionHandler-vamshi,60000,1377263788019-38] because the executor is
> missing. Is this process shutting down?
> 2013-08-23 18:56:36,906 DEBUG
> org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
> org.apache.hadoop.hbase.catalog.CatalogTracker@451415c8
> 2013-08-23 18:56:36,906 INFO
> org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor:
> vamshi,60000,1377263788019.timeoutMonitor exiting
> 2013-08-23 18:56:36,906 INFO
> org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater:
> vamshi,60000,1377263788019.timerUpdater exiting
> 2013-08-23 18:56:36,907 INFO
> org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor:
> vamshi,60000,1377263788019.splitLogManagerTimeoutMonitor exiting
> 2013-08-23 18:56:36,910 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
> region=05e30711673614f6b41a364c76f3f05f
> 2013-08-23 18:56:36,911 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
> for
>
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> destination server is vamshi,60020,1377263789273
> 2013-08-23 18:56:36,911 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
> plan was found (or we are ignoring an existing plan) for
>
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> so generated a random one;
>
> hri=scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.,
> src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
> servers
> 2013-08-23 18:56:36,911 ERROR
> org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
> [ClosedRegionHandler-vamshi,60000,1377263788019-39] because the executor is
> missing. Is this process shutting down?
> 2013-08-23 18:56:36,912 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
> ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for
> /hbase/unassigned/d476f8442ce31de90b60080b74daf47f
> 2013-08-23 18:56:36,912 INFO org.apache.hadoop.hbase.util.RetryCounter:
> Sleeping 2000ms before retry #1...
> 2013-08-23 18:56:36,914 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x140ab519b0f0000 closed
> 2013-08-23 18:56:36,914 INFO org.apache.hadoop.hbase.master.HMaster:
> HMaster main thread exiting
> 2013-08-23 18:56:36,914 ERROR
> org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
> java.lang.RuntimeException: HMaster Aborted
> at
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
> at
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2100)
>
>
>
> My hbase-site.xml :
>
> <configuration>
> <property>
> <name>hbase.rootdir</name>
>
> <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/hbstmp/</value>
> </property>
>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.master</name>
> <value>vamshi</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.clientPort</name>
> <value>2181</value>
> </property>
>
>
> <property>
> <name>hbase.hregion.max.filesize</name>
> <value>10485760</value>
> </property>
>
>
>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>vamshi</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/zkptmp</value>
> </property>
>
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>1024</value>
> </property>
>
> <property>
> <name>hbase.coprocessor.user.region.classes</name>
> <value>com.bil.coproc.ColumnAggregationEndpoint</value>
> </property>
> </configuration>
>
>
>
>
> On Fri, Aug 23, 2013 at 7:00 PM, Frank Chow <zh...@gmail.com>
> wrote:
>
> > Hi,
> > You may should check if the compact is on. If data size in a region is
> max
> > than the limition, region will split and balance after a major
> > compaction(Usually occur automatically).
> > You can manually by run the compact operaction by the shell commond:
> > compact <tableName>, or major_compact <tableName>
> >
> >
> >
> >
> > Frank Chow
>
>
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *
>
--
*Regards*
*
Vamshi Krishna
*
Re: Will hbase automatically distribute the data across region servers or NOT..??
Posted by Dhaval Shah <pr...@yahoo.co.in>.
Vamshi, max value for hbase.hregion.max.filesize to 10MB seems too small. Did you mean 10GB?
Regards,
Dhaval
________________________________
From: Vamshi Krishna <va...@gmail.com>
To: user@hbase.apache.org; zhoushuaifeng <zh...@gmail.com>
Sent: Friday, 23 August 2013 9:38 AM
Subject: Re: Will hbase automatically distribute the data across region servers or NOT..??
Thanks for the clarifications.
I am using hbase-0.94.10 and zookeepr-3.4.5
But I am running into different issues .
I set hbase.hregion.max.filesize to 10Mb and i am inserting 10 million
rows in to hbase table. During the insertion after some time, suddenly
master is going down. I don't know what is the reason for such peculiar
behavior.
I found in master log below content and not able to make out what exactly
the mistake. Please somebody help.
master-log:
2013-08-23 18:56:36,865 FATAL org.apache.hadoop.hbase.master.HMaster:
Master server abort: loaded coprocessors are: []
2013-08-23 18:56:36,866 FATAL org.apache.hadoop.hbase.master.HMaster:
Unexpected state :
scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
Cannot transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state :
scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
Cannot transit it to OFFLINE.
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,867 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2013-08-23 18:56:36,867 DEBUG org.apache.hadoop.hbase.master.HMaster:
Stopping service threads
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60000
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60000: exiting
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60000: exiting
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server listener on 60000
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 2 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 1 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster$2:
vamshi,60000,1377263788019-BalancerChore exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster:
Stopping infoServer
2013-08-23 18:56:36,873 INFO
org.apache.hadoop.hbase.master.cleaner.HFileCleaner:
master-vamshi,60000,1377263788019.archivedHFileCleaner exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
vamshi,60000,1377263788019-CatalogJanitor exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 0 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.mortbay.log: Stopped
SelectChannelConnector@0.0.0.0:60010
2013-08-23 18:56:36,874 INFO
org.apache.hadoop.hbase.master.cleaner.LogCleaner:
master-vamshi,60000,1377263788019.oldLogCleaner exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2013-08-23 18:56:36,876 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60000: exiting
2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x140ab519b0f0000 Unable to set watcher on znode
(/hbase/unassigned/05e30711673614f6b41a364c76f3f05f)
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:271)
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,876 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x08b8,1377264147374.39794b7deea3203fc260756f5038d6f8.
state=OFFLINE, ts=1377264396802, server=null
2013-08-23 18:56:36,876 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x140ab519b0f0000 Unable to get data of znode
/hbase/unassigned/d476f8442ce31de90b60080b74daf47f
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:746)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.getDataNoWatch(ZKAssign.java:904)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:283)
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,877 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
state=OFFLINE, ts=1377264396862, server=null
2013-08-23 18:56:36,877 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
state=OFFLINE, ts=1377264396813, server=null
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
region=d476f8442ce31de90b60080b74daf47f
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
destination server is vamshi,60020,1377263789273
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
so generated a random one;
hri=scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.,
src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
servers
2013-08-23 18:56:36,882 ERROR
org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
[ClosedRegionHandler-vamshi,60000,1377263788019-38] because the executor is
missing. Is this process shutting down?
2013-08-23 18:56:36,906 DEBUG
org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
org.apache.hadoop.hbase.catalog.CatalogTracker@451415c8
2013-08-23 18:56:36,906 INFO
org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor:
vamshi,60000,1377263788019.timeoutMonitor exiting
2013-08-23 18:56:36,906 INFO
org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater:
vamshi,60000,1377263788019.timerUpdater exiting
2013-08-23 18:56:36,907 INFO
org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor:
vamshi,60000,1377263788019.splitLogManagerTimeoutMonitor exiting
2013-08-23 18:56:36,910 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
region=05e30711673614f6b41a364c76f3f05f
2013-08-23 18:56:36,911 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
destination server is vamshi,60020,1377263789273
2013-08-23 18:56:36,911 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
so generated a random one;
hri=scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.,
src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
servers
2013-08-23 18:56:36,911 ERROR
org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
[ClosedRegionHandler-vamshi,60000,1377263788019-39] because the executor is
missing. Is this process shutting down?
2013-08-23 18:56:36,912 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for
/hbase/unassigned/d476f8442ce31de90b60080b74daf47f
2013-08-23 18:56:36,912 INFO org.apache.hadoop.hbase.util.RetryCounter:
Sleeping 2000ms before retry #1...
2013-08-23 18:56:36,914 INFO org.apache.zookeeper.ZooKeeper: Session:
0x140ab519b0f0000 closed
2013-08-23 18:56:36,914 INFO org.apache.hadoop.hbase.master.HMaster:
HMaster main thread exiting
2013-08-23 18:56:36,914 ERROR
org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: HMaster Aborted
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2100)
My hbase-site.xml :
<configuration>
<property>
<name>hbase.rootdir</name>
<value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/hbstmp/</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.master</name>
<value>vamshi</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.hregion.max.filesize</name>
<value>10485760</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>vamshi</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/zkptmp</value>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>1024</value>
</property>
<property>
<name>hbase.coprocessor.user.region.classes</name>
<value>com.bil.coproc.ColumnAggregationEndpoint</value>
</property>
</configuration>
On Fri, Aug 23, 2013 at 7:00 PM, Frank Chow <zh...@gmail.com> wrote:
> Hi,
> You may should check if the compact is on. If data size in a region is max
> than the limition, region will split and balance after a major
> compaction(Usually occur automatically).
> You can manually by run the compact operaction by the shell commond:
> compact <tableName>, or major_compact <tableName>
>
>
>
>
> Frank Chow
--
*Regards*
*
Vamshi Krishna
*
Re: Will hbase automatically distribute the data across region
servers or NOT..??
Posted by Vamshi Krishna <va...@gmail.com>.
Thanks for the clarifications.
I am using hbase-0.94.10 and zookeepr-3.4.5
But I am running into different issues .
I set hbase.hregion.max.filesize to 10Mb and i am inserting 10 million
rows in to hbase table. During the insertion after some time, suddenly
master is going down. I don't know what is the reason for such peculiar
behavior.
I found in master log below content and not able to make out what exactly
the mistake. Please somebody help.
master-log:
2013-08-23 18:56:36,865 FATAL org.apache.hadoop.hbase.master.HMaster:
Master server abort: loaded coprocessors are: []
2013-08-23 18:56:36,866 FATAL org.apache.hadoop.hbase.master.HMaster:
Unexpected state :
scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
Cannot transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state :
scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
Cannot transit it to OFFLINE.
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,867 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2013-08-23 18:56:36,867 DEBUG org.apache.hadoop.hbase.master.HMaster:
Stopping service threads
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60000
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60000: exiting
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60000: exiting
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server listener on 60000
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 2 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 1 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster$2:
vamshi,60000,1377263788019-BalancerChore exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster:
Stopping infoServer
2013-08-23 18:56:36,873 INFO
org.apache.hadoop.hbase.master.cleaner.HFileCleaner:
master-vamshi,60000,1377263788019.archivedHFileCleaner exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
vamshi,60000,1377263788019-CatalogJanitor exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 0 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.mortbay.log: Stopped
SelectChannelConnector@0.0.0.0:60010
2013-08-23 18:56:36,874 INFO
org.apache.hadoop.hbase.master.cleaner.LogCleaner:
master-vamshi,60000,1377263788019.oldLogCleaner exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2013-08-23 18:56:36,876 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60000: exiting
2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x140ab519b0f0000 Unable to set watcher on znode
(/hbase/unassigned/05e30711673614f6b41a364c76f3f05f)
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:271)
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,876 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x08b8,1377264147374.39794b7deea3203fc260756f5038d6f8.
state=OFFLINE, ts=1377264396802, server=null
2013-08-23 18:56:36,876 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x140ab519b0f0000 Unable to get data of znode
/hbase/unassigned/d476f8442ce31de90b60080b74daf47f
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:746)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.getDataNoWatch(ZKAssign.java:904)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:283)
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,877 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
state=OFFLINE, ts=1377264396862, server=null
2013-08-23 18:56:36,877 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
state=OFFLINE, ts=1377264396813, server=null
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
region=d476f8442ce31de90b60080b74daf47f
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
destination server is vamshi,60020,1377263789273
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
so generated a random one;
hri=scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.,
src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
servers
2013-08-23 18:56:36,882 ERROR
org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
[ClosedRegionHandler-vamshi,60000,1377263788019-38] because the executor is
missing. Is this process shutting down?
2013-08-23 18:56:36,906 DEBUG
org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
org.apache.hadoop.hbase.catalog.CatalogTracker@451415c8
2013-08-23 18:56:36,906 INFO
org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor:
vamshi,60000,1377263788019.timeoutMonitor exiting
2013-08-23 18:56:36,906 INFO
org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater:
vamshi,60000,1377263788019.timerUpdater exiting
2013-08-23 18:56:36,907 INFO
org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor:
vamshi,60000,1377263788019.splitLogManagerTimeoutMonitor exiting
2013-08-23 18:56:36,910 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
region=05e30711673614f6b41a364c76f3f05f
2013-08-23 18:56:36,911 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
destination server is vamshi,60020,1377263789273
2013-08-23 18:56:36,911 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
so generated a random one;
hri=scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.,
src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
servers
2013-08-23 18:56:36,911 ERROR
org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
[ClosedRegionHandler-vamshi,60000,1377263788019-39] because the executor is
missing. Is this process shutting down?
2013-08-23 18:56:36,912 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for
/hbase/unassigned/d476f8442ce31de90b60080b74daf47f
2013-08-23 18:56:36,912 INFO org.apache.hadoop.hbase.util.RetryCounter:
Sleeping 2000ms before retry #1...
2013-08-23 18:56:36,914 INFO org.apache.zookeeper.ZooKeeper: Session:
0x140ab519b0f0000 closed
2013-08-23 18:56:36,914 INFO org.apache.hadoop.hbase.master.HMaster:
HMaster main thread exiting
2013-08-23 18:56:36,914 ERROR
org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: HMaster Aborted
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2100)
My hbase-site.xml :
<configuration>
<property>
<name>hbase.rootdir</name>
<value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/hbstmp/</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.master</name>
<value>vamshi</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.hregion.max.filesize</name>
<value>10485760</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>vamshi</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/zkptmp</value>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>1024</value>
</property>
<property>
<name>hbase.coprocessor.user.region.classes</name>
<value>com.bil.coproc.ColumnAggregationEndpoint</value>
</property>
</configuration>
On Fri, Aug 23, 2013 at 7:00 PM, Frank Chow <zh...@gmail.com> wrote:
> Hi,
> You may should check if the compact is on. If data size in a region is max
> than the limition, region will split and balance after a major
> compaction(Usually occur automatically).
> You can manually by run the compact operaction by the shell commond:
> compact <tableName>, or major_compact <tableName>
>
>
>
>
> Frank Chow
--
*Regards*
*
Vamshi Krishna
*
Re: Will hbase automatically distribute the data across region servers or NOT..??
Posted by Frank Chow <zh...@gmail.com>.
Hi,
You may should check if the compact is on. If data size in a region is max than the limition, region will split and balance after a major compaction(Usually occur automatically).
You can manually by run the compact operaction by the shell commond: compact <tableName>, or major_compact <tableName>
Frank Chow
Re: Will hbase automatically distribute the data across region
servers or NOT..??
Posted by Stack <st...@duboce.net>.
On Fri, Aug 23, 2013 at 4:47 AM, Vamshi Krishna <va...@gmail.com>wrote:
> Hi all,
> I set up 2 node hbase cluster
Two nodes is too small of a cluster to use for making deduction about how
hbase behaves.
St.Ack