You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by ch huang <ju...@gmail.com> on 2013/07/04 07:21:09 UTC
unbalance number fo region in cluster
i insert 2GB data into my hbase cluster ,but i find the region is not
balance,is insert operation always cause the region number different on
region server?
here is my 60010 output
Table Regions Name Region Server Start Key End Key
demo2,,1372914631698.f42281ec44d113df3e644d5ecdb35366.
CH36:60030<http://ch36:60030/>
rows23703 demo2,rows23703,1372914684582.c3373791daec95b849ba2908c517e045.
CH35:60030 <http://ch35:60030/> rows23703 rows347161
demo2,rows347161,1372914714248.dd9a6c39f65ad4a879638e1bfec76016.
CH34:60030<http://ch34:60030/>
rows347161 rows428378
demo2,rows428378,1372914745946.41a4669f36bc99ef1592b2df13c0a948.
CH36:60030<http://ch36:60030/>
rows428378 rows512589
demo2,rows512589,1372914776151.a4363bdd7403633c5775931ab32a7da8.
CH36:60030<http://ch36:60030/>
rows512589 rows598644
demo2,rows598644,1372914806469.729f52018aac909e8080f00b8d698e2d.
CH36:60030<http://ch36:60030/>
rows598644 rows684871
demo2,rows684871,1372914836812.d9cb1760fe3302500034913ff813e0f8.
CH35:60030<http://ch35:60030/>
rows684871 rows771559
demo2,rows771559,1372914836812.ac16ff1404cff5c0db3d2e3b0661ffcd.
CH36:60030<http://ch36:60030/>
rows771559 Regions by Region Server Region Server Region Count
http://CH35:60030/ 2 http://CH36:60030/ 5 http://CH34:60030/ 1
Re: unbalance number fo region in cluster
Posted by ch huang <ju...@gmail.com>.
No regions in transition.
region is default size 256MB
On Thu, Jul 4, 2013 at 9:16 PM, Jean-Marc Spaggiari <jean-marc@spaggiari.org
> wrote:
> Can you please also checked on the HBase Web UI?
> Do you have regions in transition?
>
> Maybe the balancer already gave te balance plan, and regions are
> moving, slowly, to their destinations? How big are those regions? I
> mean, size in MB/GB and number of rows?
>
>
> JM
>
> 2013/7/4 Ted Yu <yu...@gmail.com>:
> > Can you check load balancer related log lines in master log and put
> them on pastebin so that root cause can be diagnosed ?
> >
> > Which HBase version are you using ?
> >
> > Thanks
> >
> > On Jul 3, 2013, at 11:44 PM, ch huang <ju...@gmail.com> wrote:
> >
> >> the load balancer process as i see is very slow i have a big table
> ,it's
> >> region distrabute is node1: 5 ,node2: 22 ,node3: 5 ,after 4 hours ,it's
> >> became
> >> node1: 6 ,node2: 20 ,node1: 6, change is very very slow!
> >>
> >> On Thu, Jul 4, 2013 at 1:54 PM, Azuryy Yu <az...@gmail.com> wrote:
> >>
> >>> I supposed load balancer is enabled and there is no pre-split.
> >>>
> >>> LB run periodicly, (5 mininutes by default), so after insert bulk data
> into
> >>> HBase, the table is unbalance but there will be balanced eventually
> after
> >>> round-robin load balancer running.
> >>>
> >>>
> >>> On Thu, Jul 4, 2013 at 1:25 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>
> >>>> Did you presplit your table ?
> >>>>
> >>>> Was load balancer enabled ?
> >>>>
> >>>> What HBase version do you use ?
> >>>>
> >>>> Thanks
> >>>>
> >>>> On Jul 3, 2013, at 10:21 PM, ch huang <ju...@gmail.com> wrote:
> >>>>
> >>>>> i insert 2GB data into my hbase cluster ,but i find the region is not
> >>>>> balance,is insert operation always cause the region number different
> on
> >>>>> region server?
> >>>>>
> >>>>> here is my 60010 output
> >>>>>
> >>>>> Table Regions Name Region Server Start Key End Key
> >>>>> demo2,,1372914631698.f42281ec44d113df3e644d5ecdb35366.
> >>>>> CH36:60030<http://ch36:60030/>
> >>>>> rows23703
> >>> demo2,rows23703,1372914684582.c3373791daec95b849ba2908c517e045.
> >>>>> CH35:60030 <http://ch35:60030/> rows23703 rows347161
> >>>>> demo2,rows347161,1372914714248.dd9a6c39f65ad4a879638e1bfec76016.
> >>>>> CH34:60030<http://ch34:60030/>
> >>>>> rows347161 rows428378
> >>>>> demo2,rows428378,1372914745946.41a4669f36bc99ef1592b2df13c0a948.
> >>>>> CH36:60030<http://ch36:60030/>
> >>>>> rows428378 rows512589
> >>>>> demo2,rows512589,1372914776151.a4363bdd7403633c5775931ab32a7da8.
> >>>>> CH36:60030<http://ch36:60030/>
> >>>>> rows512589 rows598644
> >>>>> demo2,rows598644,1372914806469.729f52018aac909e8080f00b8d698e2d.
> >>>>> CH36:60030<http://ch36:60030/>
> >>>>> rows598644 rows684871
> >>>>> demo2,rows684871,1372914836812.d9cb1760fe3302500034913ff813e0f8.
> >>>>> CH35:60030<http://ch35:60030/>
> >>>>> rows684871 rows771559
> >>>>> demo2,rows771559,1372914836812.ac16ff1404cff5c0db3d2e3b0661ffcd.
> >>>>> CH36:60030<http://ch36:60030/>
> >>>>> rows771559 Regions by Region Server Region Server Region Count
> >>>>> http://CH35:60030/ <http://ch35:60030/> <http://ch35:60030/> 2
> http://CH36:60030/ <http://ch36:60030/><http://ch36:60030/>5
> >>> http://CH34:60030/ <http://ch34:60030/> <http://ch34:60030/> 1
> >>>
>
Re: unbalance number fo region in cluster
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Can you please also checked on the HBase Web UI?
Do you have regions in transition?
Maybe the balancer already gave te balance plan, and regions are
moving, slowly, to their destinations? How big are those regions? I
mean, size in MB/GB and number of rows?
JM
2013/7/4 Ted Yu <yu...@gmail.com>:
> Can you check load balancer related log lines in master log and put them on pastebin so that root cause can be diagnosed ?
>
> Which HBase version are you using ?
>
> Thanks
>
> On Jul 3, 2013, at 11:44 PM, ch huang <ju...@gmail.com> wrote:
>
>> the load balancer process as i see is very slow i have a big table ,it's
>> region distrabute is node1: 5 ,node2: 22 ,node3: 5 ,after 4 hours ,it's
>> became
>> node1: 6 ,node2: 20 ,node1: 6, change is very very slow!
>>
>> On Thu, Jul 4, 2013 at 1:54 PM, Azuryy Yu <az...@gmail.com> wrote:
>>
>>> I supposed load balancer is enabled and there is no pre-split.
>>>
>>> LB run periodicly, (5 mininutes by default), so after insert bulk data into
>>> HBase, the table is unbalance but there will be balanced eventually after
>>> round-robin load balancer running.
>>>
>>>
>>> On Thu, Jul 4, 2013 at 1:25 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Did you presplit your table ?
>>>>
>>>> Was load balancer enabled ?
>>>>
>>>> What HBase version do you use ?
>>>>
>>>> Thanks
>>>>
>>>> On Jul 3, 2013, at 10:21 PM, ch huang <ju...@gmail.com> wrote:
>>>>
>>>>> i insert 2GB data into my hbase cluster ,but i find the region is not
>>>>> balance,is insert operation always cause the region number different on
>>>>> region server?
>>>>>
>>>>> here is my 60010 output
>>>>>
>>>>> Table Regions Name Region Server Start Key End Key
>>>>> demo2,,1372914631698.f42281ec44d113df3e644d5ecdb35366.
>>>>> CH36:60030<http://ch36:60030/>
>>>>> rows23703
>>> demo2,rows23703,1372914684582.c3373791daec95b849ba2908c517e045.
>>>>> CH35:60030 <http://ch35:60030/> rows23703 rows347161
>>>>> demo2,rows347161,1372914714248.dd9a6c39f65ad4a879638e1bfec76016.
>>>>> CH34:60030<http://ch34:60030/>
>>>>> rows347161 rows428378
>>>>> demo2,rows428378,1372914745946.41a4669f36bc99ef1592b2df13c0a948.
>>>>> CH36:60030<http://ch36:60030/>
>>>>> rows428378 rows512589
>>>>> demo2,rows512589,1372914776151.a4363bdd7403633c5775931ab32a7da8.
>>>>> CH36:60030<http://ch36:60030/>
>>>>> rows512589 rows598644
>>>>> demo2,rows598644,1372914806469.729f52018aac909e8080f00b8d698e2d.
>>>>> CH36:60030<http://ch36:60030/>
>>>>> rows598644 rows684871
>>>>> demo2,rows684871,1372914836812.d9cb1760fe3302500034913ff813e0f8.
>>>>> CH35:60030<http://ch35:60030/>
>>>>> rows684871 rows771559
>>>>> demo2,rows771559,1372914836812.ac16ff1404cff5c0db3d2e3b0661ffcd.
>>>>> CH36:60030<http://ch36:60030/>
>>>>> rows771559 Regions by Region Server Region Server Region Count
>>>>> http://CH35:60030/ <http://ch35:60030/> 2 http://CH36:60030/<http://ch36:60030/>5
>>> http://CH34:60030/ <http://ch34:60030/> 1
>>>
Re: unbalance number fo region in cluster
Posted by Ted Yu <yu...@gmail.com>.
2013-07-05 10:11:24,894 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing. servers=2 regions=145 average=72.5 mostloaded=73
leastloaded=72
Balancer saw 2 servers which were balanced - in terms of number of regions.
Where did CH36 go ?
Cheers
On Thu, Jul 4, 2013 at 7:16 PM, ch huang <ju...@gmail.com> wrote:
> i see lot's of skip balance info,is balancer refuse to work?why?
>
> 2013-07-05 09:56:25,121 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> DELETING region
> hdfs://CH22:9000/hbaseroot/demoK/febcd37c57994604c8e218d8eb9c75c2
> 2013-07-05 09:56:25,130 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
> Deleted region
> demoK,rows720752,1372989269181.febcd37c57994604c8e218d8eb9c75c2. from META
> 2013-07-05 09:56:25,130 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
> Scanned 151 catalog row(s) and gc'd 8 unreferenced parent region(s)
> 2013-07-05 10:01:24,894 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
> Server information: CH34,60020,1372899976453=72,
> CH35,60020,1372899981567=73
> 2013-07-05 10:01:24,894 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing. servers=2 regions=145 average=72.5 mostloaded=73
> leastloaded=72
> 2013-07-05 10:01:24,962 DEBUG
> org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 143 catalog row(s)
> and gc'd 0 unreferenced parent region(s)
> 2013-07-05 10:06:24,895 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
> Server information: CH34,60020,1372899976453=72,
> CH35,60020,1372899981567=73
> 2013-07-05 10:06:24,895 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing. servers=2 regions=145 average=72.5 mostloaded=73
> leastloaded=72
> 2013-07-05 10:06:24,964 DEBUG
> org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 143 catalog row(s)
> and gc'd 0 unreferenced parent region(s)
> 2013-07-05 10:10:49,441 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
> Scanning .META. starting at row=jerry_cm,,00000000000000 for max=2147483647
> rows
> 2013-07-05 10:10:49,453 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> The connection to hconnection-0x3fa2814501005d has been closed.
> 2013-07-05 10:11:07,211 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
> Scanning .META. starting at row=dmp_pageview,,00000000000000 for max=10
> rows
> 2013-07-05 10:11:07,213 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Cached location for
> dmp_pageview,,1372758866561.364a36024f18bc6047a0bf7158b4c744. is CH34:60020
> 2013-07-05 10:11:07,214 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> The connection to hconnection-0x3fa2814501005d has been closed.
> 2013-07-05 10:11:07,215 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
> Scanning .META. starting at row=dmp_pageview,,00000000000000 for
> max=2147483647 rows
> 2013-07-05 10:11:07,219 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> The connection to hconnection-0x3fa2814501005d has been closed.
> 2013-07-05 10:11:13,176 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
> Scanning .META. starting at row=usertable,,00000000000000 for max=10 rows
> 2013-07-05 10:11:13,179 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Cached location for
> usertable,,1372923221432.e2c013858f08c420c9a6c86b99068f75. is CH35:60020
> 2013-07-05 10:11:13,179 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Cached location for
>
> usertable,user2035056063890669682,1372923221432.833e4e71534caac4e34371819e42ca20.
> is CH34:60020
> 2013-07-05 10:11:13,179 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Cached location for
>
> usertable,user3071366660001926980,1372923194708.da128be082d51ec5ab1233df63aabb46.
> is CH34:60020
> 2013-07-05 10:11:13,180 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Cached location for
>
> usertable,user4104227499385169775,1372923194708.97d6138333c72839ce22adae4940e3d5.
> is CH34:60020
> 2013-07-05 10:11:13,180 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Cached location for
>
> usertable,user5139370204013525946,1372923209834.e788429ec0b1e032f4b309337c590732.
> is CH34:60020
> 2013-07-05 10:11:13,180 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Cached location for
>
> usertable,user6178888326785369277,1372923209834.ed841132313f30d51c82ad9f8e3e71c5.
> is CH34:60020
> 2013-07-05 10:11:13,180 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Cached location for
>
> usertable,user7219194014980283263,1372923217190.7a4f56a2e754a707b80dc4d4689f54e7.
> is CH34:60020
> 2013-07-05 10:11:13,181 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Cached location for
>
> usertable,user825931901641799999,1372923217190.b6275359a308360d59931b974d096c45.
> is CH34:60020
> 2013-07-05 10:11:13,182 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> The connection to hconnection-0x3fa2814501005d has been closed.
> 2013-07-05 10:11:13,183 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
> Scanning .META. starting at row=usertable,,00000000000000 for
> max=2147483647 rows
> 2013-07-05 10:11:13,188 DEBUG
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> The connection to hconnection-0x3fa2814501005d has been closed.
> 2013-07-05 10:11:24,894 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
> Server information: CH34,60020,1372899976453=72,
> CH35,60020,1372899981567=73
> 2013-07-05 10:11:24,894 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing. servers=2 regions=145 average=72.5 mostloaded=73
> leastloaded=72
> 2013-07-05 10:11:24,962 DEBUG
> org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 143 catalog row(s)
> and gc'd 0 unreferenced parent region(s)
>
>
> On Thu, Jul 4, 2013 at 5:20 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Can you check load balancer related log lines in master log and put them
> > on pastebin so that root cause can be diagnosed ?
> >
> > Which HBase version are you using ?
> >
> > Thanks
> >
> > On Jul 3, 2013, at 11:44 PM, ch huang <ju...@gmail.com> wrote:
> >
> > > the load balancer process as i see is very slow i have a big table
> ,it's
> > > region distrabute is node1: 5 ,node2: 22 ,node3: 5 ,after 4 hours
> ,it's
> > > became
> > > node1: 6 ,node2: 20 ,node1: 6, change is very very slow!
> > >
> > > On Thu, Jul 4, 2013 at 1:54 PM, Azuryy Yu <az...@gmail.com> wrote:
> > >
> > >> I supposed load balancer is enabled and there is no pre-split.
> > >>
> > >> LB run periodicly, (5 mininutes by default), so after insert bulk data
> > into
> > >> HBase, the table is unbalance but there will be balanced eventually
> > after
> > >> round-robin load balancer running.
> > >>
> > >>
> > >> On Thu, Jul 4, 2013 at 1:25 PM, Ted Yu <yu...@gmail.com> wrote:
> > >>
> > >>> Did you presplit your table ?
> > >>>
> > >>> Was load balancer enabled ?
> > >>>
> > >>> What HBase version do you use ?
> > >>>
> > >>> Thanks
> > >>>
> > >>> On Jul 3, 2013, at 10:21 PM, ch huang <ju...@gmail.com> wrote:
> > >>>
> > >>>> i insert 2GB data into my hbase cluster ,but i find the region is
> not
> > >>>> balance,is insert operation always cause the region number different
> > on
> > >>>> region server?
> > >>>>
> > >>>> here is my 60010 output
> > >>>>
> > >>>> Table Regions Name Region Server Start Key End Key
> > >>>> demo2,,1372914631698.f42281ec44d113df3e644d5ecdb35366.
> > >>>> CH36:60030<http://ch36:60030/>
> > >>>> rows23703
> > >> demo2,rows23703,1372914684582.c3373791daec95b849ba2908c517e045.
> > >>>> CH35:60030 <http://ch35:60030/> rows23703 rows347161
> > >>>> demo2,rows347161,1372914714248.dd9a6c39f65ad4a879638e1bfec76016.
> > >>>> CH34:60030<http://ch34:60030/>
> > >>>> rows347161 rows428378
> > >>>> demo2,rows428378,1372914745946.41a4669f36bc99ef1592b2df13c0a948.
> > >>>> CH36:60030<http://ch36:60030/>
> > >>>> rows428378 rows512589
> > >>>> demo2,rows512589,1372914776151.a4363bdd7403633c5775931ab32a7da8.
> > >>>> CH36:60030<http://ch36:60030/>
> > >>>> rows512589 rows598644
> > >>>> demo2,rows598644,1372914806469.729f52018aac909e8080f00b8d698e2d.
> > >>>> CH36:60030<http://ch36:60030/>
> > >>>> rows598644 rows684871
> > >>>> demo2,rows684871,1372914836812.d9cb1760fe3302500034913ff813e0f8.
> > >>>> CH35:60030<http://ch35:60030/>
> > >>>> rows684871 rows771559
> > >>>> demo2,rows771559,1372914836812.ac16ff1404cff5c0db3d2e3b0661ffcd.
> > >>>> CH36:60030<http://ch36:60030/>
> > >>>> rows771559 Regions by Region Server Region Server Region Count
> > >>>> http://CH35:60030/ <http://ch35:60030/> <http://ch35:60030/> 2
> > http://CH36:60030/ <http://ch36:60030/><http://ch36:60030/>5
> > >> http://CH34:60030/ <http://ch34:60030/> <http://ch34:60030/> 1
> > >>
> >
>
Re: unbalance number fo region in cluster
Posted by ch huang <ju...@gmail.com>.
i see lot's of skip balance info,is balancer refuse to work?why?
2013-07-05 09:56:25,121 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
DELETING region
hdfs://CH22:9000/hbaseroot/demoK/febcd37c57994604c8e218d8eb9c75c2
2013-07-05 09:56:25,130 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Deleted region
demoK,rows720752,1372989269181.febcd37c57994604c8e218d8eb9c75c2. from META
2013-07-05 09:56:25,130 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 151 catalog row(s) and gc'd 8 unreferenced parent region(s)
2013-07-05 10:01:24,894 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information: CH34,60020,1372899976453=72, CH35,60020,1372899981567=73
2013-07-05 10:01:24,894 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing. servers=2 regions=145 average=72.5 mostloaded=73
leastloaded=72
2013-07-05 10:01:24,962 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 143 catalog row(s)
and gc'd 0 unreferenced parent region(s)
2013-07-05 10:06:24,895 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information: CH34,60020,1372899976453=72, CH35,60020,1372899981567=73
2013-07-05 10:06:24,895 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing. servers=2 regions=145 average=72.5 mostloaded=73
leastloaded=72
2013-07-05 10:06:24,964 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 143 catalog row(s)
and gc'd 0 unreferenced parent region(s)
2013-07-05 10:10:49,441 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
Scanning .META. starting at row=jerry_cm,,00000000000000 for max=2147483647
rows
2013-07-05 10:10:49,453 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to hconnection-0x3fa2814501005d has been closed.
2013-07-05 10:11:07,211 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
Scanning .META. starting at row=dmp_pageview,,00000000000000 for max=10 rows
2013-07-05 10:11:07,213 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for
dmp_pageview,,1372758866561.364a36024f18bc6047a0bf7158b4c744. is CH34:60020
2013-07-05 10:11:07,214 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to hconnection-0x3fa2814501005d has been closed.
2013-07-05 10:11:07,215 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
Scanning .META. starting at row=dmp_pageview,,00000000000000 for
max=2147483647 rows
2013-07-05 10:11:07,219 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to hconnection-0x3fa2814501005d has been closed.
2013-07-05 10:11:13,176 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
Scanning .META. starting at row=usertable,,00000000000000 for max=10 rows
2013-07-05 10:11:13,179 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for
usertable,,1372923221432.e2c013858f08c420c9a6c86b99068f75. is CH35:60020
2013-07-05 10:11:13,179 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for
usertable,user2035056063890669682,1372923221432.833e4e71534caac4e34371819e42ca20.
is CH34:60020
2013-07-05 10:11:13,179 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for
usertable,user3071366660001926980,1372923194708.da128be082d51ec5ab1233df63aabb46.
is CH34:60020
2013-07-05 10:11:13,180 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for
usertable,user4104227499385169775,1372923194708.97d6138333c72839ce22adae4940e3d5.
is CH34:60020
2013-07-05 10:11:13,180 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for
usertable,user5139370204013525946,1372923209834.e788429ec0b1e032f4b309337c590732.
is CH34:60020
2013-07-05 10:11:13,180 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for
usertable,user6178888326785369277,1372923209834.ed841132313f30d51c82ad9f8e3e71c5.
is CH34:60020
2013-07-05 10:11:13,180 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for
usertable,user7219194014980283263,1372923217190.7a4f56a2e754a707b80dc4d4689f54e7.
is CH34:60020
2013-07-05 10:11:13,181 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for
usertable,user825931901641799999,1372923217190.b6275359a308360d59931b974d096c45.
is CH34:60020
2013-07-05 10:11:13,182 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to hconnection-0x3fa2814501005d has been closed.
2013-07-05 10:11:13,183 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
Scanning .META. starting at row=usertable,,00000000000000 for
max=2147483647 rows
2013-07-05 10:11:13,188 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to hconnection-0x3fa2814501005d has been closed.
2013-07-05 10:11:24,894 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information: CH34,60020,1372899976453=72, CH35,60020,1372899981567=73
2013-07-05 10:11:24,894 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing. servers=2 regions=145 average=72.5 mostloaded=73
leastloaded=72
2013-07-05 10:11:24,962 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 143 catalog row(s)
and gc'd 0 unreferenced parent region(s)
On Thu, Jul 4, 2013 at 5:20 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you check load balancer related log lines in master log and put them
> on pastebin so that root cause can be diagnosed ?
>
> Which HBase version are you using ?
>
> Thanks
>
> On Jul 3, 2013, at 11:44 PM, ch huang <ju...@gmail.com> wrote:
>
> > the load balancer process as i see is very slow i have a big table ,it's
> > region distrabute is node1: 5 ,node2: 22 ,node3: 5 ,after 4 hours ,it's
> > became
> > node1: 6 ,node2: 20 ,node1: 6, change is very very slow!
> >
> > On Thu, Jul 4, 2013 at 1:54 PM, Azuryy Yu <az...@gmail.com> wrote:
> >
> >> I supposed load balancer is enabled and there is no pre-split.
> >>
> >> LB run periodicly, (5 mininutes by default), so after insert bulk data
> into
> >> HBase, the table is unbalance but there will be balanced eventually
> after
> >> round-robin load balancer running.
> >>
> >>
> >> On Thu, Jul 4, 2013 at 1:25 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >>> Did you presplit your table ?
> >>>
> >>> Was load balancer enabled ?
> >>>
> >>> What HBase version do you use ?
> >>>
> >>> Thanks
> >>>
> >>> On Jul 3, 2013, at 10:21 PM, ch huang <ju...@gmail.com> wrote:
> >>>
> >>>> i insert 2GB data into my hbase cluster ,but i find the region is not
> >>>> balance,is insert operation always cause the region number different
> on
> >>>> region server?
> >>>>
> >>>> here is my 60010 output
> >>>>
> >>>> Table Regions Name Region Server Start Key End Key
> >>>> demo2,,1372914631698.f42281ec44d113df3e644d5ecdb35366.
> >>>> CH36:60030<http://ch36:60030/>
> >>>> rows23703
> >> demo2,rows23703,1372914684582.c3373791daec95b849ba2908c517e045.
> >>>> CH35:60030 <http://ch35:60030/> rows23703 rows347161
> >>>> demo2,rows347161,1372914714248.dd9a6c39f65ad4a879638e1bfec76016.
> >>>> CH34:60030<http://ch34:60030/>
> >>>> rows347161 rows428378
> >>>> demo2,rows428378,1372914745946.41a4669f36bc99ef1592b2df13c0a948.
> >>>> CH36:60030<http://ch36:60030/>
> >>>> rows428378 rows512589
> >>>> demo2,rows512589,1372914776151.a4363bdd7403633c5775931ab32a7da8.
> >>>> CH36:60030<http://ch36:60030/>
> >>>> rows512589 rows598644
> >>>> demo2,rows598644,1372914806469.729f52018aac909e8080f00b8d698e2d.
> >>>> CH36:60030<http://ch36:60030/>
> >>>> rows598644 rows684871
> >>>> demo2,rows684871,1372914836812.d9cb1760fe3302500034913ff813e0f8.
> >>>> CH35:60030<http://ch35:60030/>
> >>>> rows684871 rows771559
> >>>> demo2,rows771559,1372914836812.ac16ff1404cff5c0db3d2e3b0661ffcd.
> >>>> CH36:60030<http://ch36:60030/>
> >>>> rows771559 Regions by Region Server Region Server Region Count
> >>>> http://CH35:60030/ <http://ch35:60030/> <http://ch35:60030/> 2
> http://CH36:60030/ <http://ch36:60030/><http://ch36:60030/>5
> >> http://CH34:60030/ <http://ch34:60030/> <http://ch34:60030/> 1
> >>
>
Re: unbalance number fo region in cluster
Posted by Ted Yu <yu...@gmail.com>.
Can you check load balancer related log lines in master log and put them on pastebin so that root cause can be diagnosed ?
Which HBase version are you using ?
Thanks
On Jul 3, 2013, at 11:44 PM, ch huang <ju...@gmail.com> wrote:
> the load balancer process as i see is very slow i have a big table ,it's
> region distrabute is node1: 5 ,node2: 22 ,node3: 5 ,after 4 hours ,it's
> became
> node1: 6 ,node2: 20 ,node1: 6, change is very very slow!
>
> On Thu, Jul 4, 2013 at 1:54 PM, Azuryy Yu <az...@gmail.com> wrote:
>
>> I supposed load balancer is enabled and there is no pre-split.
>>
>> LB run periodicly, (5 mininutes by default), so after insert bulk data into
>> HBase, the table is unbalance but there will be balanced eventually after
>> round-robin load balancer running.
>>
>>
>> On Thu, Jul 4, 2013 at 1:25 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Did you presplit your table ?
>>>
>>> Was load balancer enabled ?
>>>
>>> What HBase version do you use ?
>>>
>>> Thanks
>>>
>>> On Jul 3, 2013, at 10:21 PM, ch huang <ju...@gmail.com> wrote:
>>>
>>>> i insert 2GB data into my hbase cluster ,but i find the region is not
>>>> balance,is insert operation always cause the region number different on
>>>> region server?
>>>>
>>>> here is my 60010 output
>>>>
>>>> Table Regions Name Region Server Start Key End Key
>>>> demo2,,1372914631698.f42281ec44d113df3e644d5ecdb35366.
>>>> CH36:60030<http://ch36:60030/>
>>>> rows23703
>> demo2,rows23703,1372914684582.c3373791daec95b849ba2908c517e045.
>>>> CH35:60030 <http://ch35:60030/> rows23703 rows347161
>>>> demo2,rows347161,1372914714248.dd9a6c39f65ad4a879638e1bfec76016.
>>>> CH34:60030<http://ch34:60030/>
>>>> rows347161 rows428378
>>>> demo2,rows428378,1372914745946.41a4669f36bc99ef1592b2df13c0a948.
>>>> CH36:60030<http://ch36:60030/>
>>>> rows428378 rows512589
>>>> demo2,rows512589,1372914776151.a4363bdd7403633c5775931ab32a7da8.
>>>> CH36:60030<http://ch36:60030/>
>>>> rows512589 rows598644
>>>> demo2,rows598644,1372914806469.729f52018aac909e8080f00b8d698e2d.
>>>> CH36:60030<http://ch36:60030/>
>>>> rows598644 rows684871
>>>> demo2,rows684871,1372914836812.d9cb1760fe3302500034913ff813e0f8.
>>>> CH35:60030<http://ch35:60030/>
>>>> rows684871 rows771559
>>>> demo2,rows771559,1372914836812.ac16ff1404cff5c0db3d2e3b0661ffcd.
>>>> CH36:60030<http://ch36:60030/>
>>>> rows771559 Regions by Region Server Region Server Region Count
>>>> http://CH35:60030/ <http://ch35:60030/> 2 http://CH36:60030/<http://ch36:60030/>5
>> http://CH34:60030/ <http://ch34:60030/> 1
>>
Re: unbalance number fo region in cluster
Posted by ch huang <ju...@gmail.com>.
the load balancer process as i see is very slow i have a big table ,it's
region distrabute is node1: 5 ,node2: 22 ,node3: 5 ,after 4 hours ,it's
became
node1: 6 ,node2: 20 ,node1: 6, change is very very slow!
On Thu, Jul 4, 2013 at 1:54 PM, Azuryy Yu <az...@gmail.com> wrote:
> I supposed load balancer is enabled and there is no pre-split.
>
> LB run periodicly, (5 mininutes by default), so after insert bulk data into
> HBase, the table is unbalance but there will be balanced eventually after
> round-robin load balancer running.
>
>
> On Thu, Jul 4, 2013 at 1:25 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Did you presplit your table ?
> >
> > Was load balancer enabled ?
> >
> > What HBase version do you use ?
> >
> > Thanks
> >
> > On Jul 3, 2013, at 10:21 PM, ch huang <ju...@gmail.com> wrote:
> >
> > > i insert 2GB data into my hbase cluster ,but i find the region is not
> > > balance,is insert operation always cause the region number different on
> > > region server?
> > >
> > > here is my 60010 output
> > >
> > > Table Regions Name Region Server Start Key End Key
> > > demo2,,1372914631698.f42281ec44d113df3e644d5ecdb35366.
> > > CH36:60030<http://ch36:60030/>
> > > rows23703
> demo2,rows23703,1372914684582.c3373791daec95b849ba2908c517e045.
> > > CH35:60030 <http://ch35:60030/> rows23703 rows347161
> > > demo2,rows347161,1372914714248.dd9a6c39f65ad4a879638e1bfec76016.
> > > CH34:60030<http://ch34:60030/>
> > > rows347161 rows428378
> > > demo2,rows428378,1372914745946.41a4669f36bc99ef1592b2df13c0a948.
> > > CH36:60030<http://ch36:60030/>
> > > rows428378 rows512589
> > > demo2,rows512589,1372914776151.a4363bdd7403633c5775931ab32a7da8.
> > > CH36:60030<http://ch36:60030/>
> > > rows512589 rows598644
> > > demo2,rows598644,1372914806469.729f52018aac909e8080f00b8d698e2d.
> > > CH36:60030<http://ch36:60030/>
> > > rows598644 rows684871
> > > demo2,rows684871,1372914836812.d9cb1760fe3302500034913ff813e0f8.
> > > CH35:60030<http://ch35:60030/>
> > > rows684871 rows771559
> > > demo2,rows771559,1372914836812.ac16ff1404cff5c0db3d2e3b0661ffcd.
> > > CH36:60030<http://ch36:60030/>
> > > rows771559 Regions by Region Server Region Server Region Count
> > > http://CH35:60030/ <http://ch35:60030/> 2 http://CH36:60030/<http://ch36:60030/>5
> http://CH34:60030/ <http://ch34:60030/> 1
> >
>
Re: unbalance number fo region in cluster
Posted by Azuryy Yu <az...@gmail.com>.
I supposed load balancer is enabled and there is no pre-split.
LB run periodicly, (5 mininutes by default), so after insert bulk data into
HBase, the table is unbalance but there will be balanced eventually after
round-robin load balancer running.
On Thu, Jul 4, 2013 at 1:25 PM, Ted Yu <yu...@gmail.com> wrote:
> Did you presplit your table ?
>
> Was load balancer enabled ?
>
> What HBase version do you use ?
>
> Thanks
>
> On Jul 3, 2013, at 10:21 PM, ch huang <ju...@gmail.com> wrote:
>
> > i insert 2GB data into my hbase cluster ,but i find the region is not
> > balance,is insert operation always cause the region number different on
> > region server?
> >
> > here is my 60010 output
> >
> > Table Regions Name Region Server Start Key End Key
> > demo2,,1372914631698.f42281ec44d113df3e644d5ecdb35366.
> > CH36:60030<http://ch36:60030/>
> > rows23703 demo2,rows23703,1372914684582.c3373791daec95b849ba2908c517e045.
> > CH35:60030 <http://ch35:60030/> rows23703 rows347161
> > demo2,rows347161,1372914714248.dd9a6c39f65ad4a879638e1bfec76016.
> > CH34:60030<http://ch34:60030/>
> > rows347161 rows428378
> > demo2,rows428378,1372914745946.41a4669f36bc99ef1592b2df13c0a948.
> > CH36:60030<http://ch36:60030/>
> > rows428378 rows512589
> > demo2,rows512589,1372914776151.a4363bdd7403633c5775931ab32a7da8.
> > CH36:60030<http://ch36:60030/>
> > rows512589 rows598644
> > demo2,rows598644,1372914806469.729f52018aac909e8080f00b8d698e2d.
> > CH36:60030<http://ch36:60030/>
> > rows598644 rows684871
> > demo2,rows684871,1372914836812.d9cb1760fe3302500034913ff813e0f8.
> > CH35:60030<http://ch35:60030/>
> > rows684871 rows771559
> > demo2,rows771559,1372914836812.ac16ff1404cff5c0db3d2e3b0661ffcd.
> > CH36:60030<http://ch36:60030/>
> > rows771559 Regions by Region Server Region Server Region Count
> > http://CH35:60030/ 2 http://CH36:60030/ 5 http://CH34:60030/ 1
>
Re: unbalance number fo region in cluster
Posted by Ted Yu <yu...@gmail.com>.
bq. CH3u4
Did you mean CDH3u4 ?
How many tables do you have in total ?
Are regions from all the tables balanced across your cluster ?
Cheers
On Wed, Jul 3, 2013 at 11:00 PM, ch huang <ju...@gmail.com> wrote:
> hi,i use CH3u4 , no presplit, balancer is enabled,thanks
>
>
>
>
> On Thu, Jul 4, 2013 at 1:25 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Did you presplit your table ?
> >
> > Was load balancer enabled ?
> >
> > What HBase version do you use ?
> >
> > Thanks
> >
> > On Jul 3, 2013, at 10:21 PM, ch huang <ju...@gmail.com> wrote:
> >
> > > i insert 2GB data into my hbase cluster ,but i find the region is not
> > > balance,is insert operation always cause the region number different on
> > > region server?
> > >
> > > here is my 60010 output
> > >
> > > Table Regions Name Region Server Start Key End Key
> > > demo2,,1372914631698.f42281ec44d113df3e644d5ecdb35366.
> > > CH36:60030<http://ch36:60030/>
> > > rows23703
> demo2,rows23703,1372914684582.c3373791daec95b849ba2908c517e045.
> > > CH35:60030 <http://ch35:60030/> rows23703 rows347161
> > > demo2,rows347161,1372914714248.dd9a6c39f65ad4a879638e1bfec76016.
> > > CH34:60030<http://ch34:60030/>
> > > rows347161 rows428378
> > > demo2,rows428378,1372914745946.41a4669f36bc99ef1592b2df13c0a948.
> > > CH36:60030<http://ch36:60030/>
> > > rows428378 rows512589
> > > demo2,rows512589,1372914776151.a4363bdd7403633c5775931ab32a7da8.
> > > CH36:60030<http://ch36:60030/>
> > > rows512589 rows598644
> > > demo2,rows598644,1372914806469.729f52018aac909e8080f00b8d698e2d.
> > > CH36:60030<http://ch36:60030/>
> > > rows598644 rows684871
> > > demo2,rows684871,1372914836812.d9cb1760fe3302500034913ff813e0f8.
> > > CH35:60030<http://ch35:60030/>
> > > rows684871 rows771559
> > > demo2,rows771559,1372914836812.ac16ff1404cff5c0db3d2e3b0661ffcd.
> > > CH36:60030<http://ch36:60030/>
> > > rows771559 Regions by Region Server Region Server Region Count
> > > http://CH35:60030/ <http://ch35:60030/> 2 http://CH36:60030/<
> http://ch36:60030/>5
> > http://CH34:60030/ <http://ch34:60030/> 1
> >
>
Re: unbalance number fo region in cluster
Posted by ch huang <ju...@gmail.com>.
hi,i use CH3u4 , no presplit, balancer is enabled,thanks
On Thu, Jul 4, 2013 at 1:25 PM, Ted Yu <yu...@gmail.com> wrote:
> Did you presplit your table ?
>
> Was load balancer enabled ?
>
> What HBase version do you use ?
>
> Thanks
>
> On Jul 3, 2013, at 10:21 PM, ch huang <ju...@gmail.com> wrote:
>
> > i insert 2GB data into my hbase cluster ,but i find the region is not
> > balance,is insert operation always cause the region number different on
> > region server?
> >
> > here is my 60010 output
> >
> > Table Regions Name Region Server Start Key End Key
> > demo2,,1372914631698.f42281ec44d113df3e644d5ecdb35366.
> > CH36:60030<http://ch36:60030/>
> > rows23703 demo2,rows23703,1372914684582.c3373791daec95b849ba2908c517e045.
> > CH35:60030 <http://ch35:60030/> rows23703 rows347161
> > demo2,rows347161,1372914714248.dd9a6c39f65ad4a879638e1bfec76016.
> > CH34:60030<http://ch34:60030/>
> > rows347161 rows428378
> > demo2,rows428378,1372914745946.41a4669f36bc99ef1592b2df13c0a948.
> > CH36:60030<http://ch36:60030/>
> > rows428378 rows512589
> > demo2,rows512589,1372914776151.a4363bdd7403633c5775931ab32a7da8.
> > CH36:60030<http://ch36:60030/>
> > rows512589 rows598644
> > demo2,rows598644,1372914806469.729f52018aac909e8080f00b8d698e2d.
> > CH36:60030<http://ch36:60030/>
> > rows598644 rows684871
> > demo2,rows684871,1372914836812.d9cb1760fe3302500034913ff813e0f8.
> > CH35:60030<http://ch35:60030/>
> > rows684871 rows771559
> > demo2,rows771559,1372914836812.ac16ff1404cff5c0db3d2e3b0661ffcd.
> > CH36:60030<http://ch36:60030/>
> > rows771559 Regions by Region Server Region Server Region Count
> > http://CH35:60030/ <http://ch35:60030/> 2 http://CH36:60030/<http://ch36:60030/>5
> http://CH34:60030/ <http://ch34:60030/> 1
>
Re: unbalance number fo region in cluster
Posted by Ted Yu <yu...@gmail.com>.
Did you presplit your table ?
Was load balancer enabled ?
What HBase version do you use ?
Thanks
On Jul 3, 2013, at 10:21 PM, ch huang <ju...@gmail.com> wrote:
> i insert 2GB data into my hbase cluster ,but i find the region is not
> balance,is insert operation always cause the region number different on
> region server?
>
> here is my 60010 output
>
> Table Regions Name Region Server Start Key End Key
> demo2,,1372914631698.f42281ec44d113df3e644d5ecdb35366.
> CH36:60030<http://ch36:60030/>
> rows23703 demo2,rows23703,1372914684582.c3373791daec95b849ba2908c517e045.
> CH35:60030 <http://ch35:60030/> rows23703 rows347161
> demo2,rows347161,1372914714248.dd9a6c39f65ad4a879638e1bfec76016.
> CH34:60030<http://ch34:60030/>
> rows347161 rows428378
> demo2,rows428378,1372914745946.41a4669f36bc99ef1592b2df13c0a948.
> CH36:60030<http://ch36:60030/>
> rows428378 rows512589
> demo2,rows512589,1372914776151.a4363bdd7403633c5775931ab32a7da8.
> CH36:60030<http://ch36:60030/>
> rows512589 rows598644
> demo2,rows598644,1372914806469.729f52018aac909e8080f00b8d698e2d.
> CH36:60030<http://ch36:60030/>
> rows598644 rows684871
> demo2,rows684871,1372914836812.d9cb1760fe3302500034913ff813e0f8.
> CH35:60030<http://ch35:60030/>
> rows684871 rows771559
> demo2,rows771559,1372914836812.ac16ff1404cff5c0db3d2e3b0661ffcd.
> CH36:60030<http://ch36:60030/>
> rows771559 Regions by Region Server Region Server Region Count
> http://CH35:60030/ 2 http://CH36:60030/ 5 http://CH34:60030/ 1