You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Thibaut_ <tb...@blue.lu> on 2008/12/17 19:31:39 UTC

Table with 80 regions having nearly no data in it

Hi,

I upgraded to the trunk version (0.19.0-dev, r726278) of hbase and I'm
pretty happy with it.

Also the lockups I had before, seemed to go away by not creating a new
instance of HTable on each update/delete/exists call. I also batched up
multiple writes to the tables (1 commit each 1000 rows), which seemed to
further reduce the load on the servers. My jobs finally run to an end,
pretty good :-)
As a side note, it would be helpfull if it would be possible not only to
insert or change rows with BatchUpdate, but also to delete rows (So I can
delete more rows at the end when I'm executing the other batched requests as
well).



But something I have noticed is that I have a table (at least one)

tobeprocessed 	{NAME => 'tobeprocessed', IS_ROOT => 'false', IS_META =>
'false', FAMILIES => [{NAME => 'data', BLOOMFILTER => 'false', COMPRESSION
=> 'NONE', VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY
=> 'false', BLOCKCACHE => 'false'}], INDEXES => []}

which spans over 70 regions, but only has about 117 rows in it (just a few
MByte). These entries are all in the last region (as I used a timestamp as
key and I just checked with a mapreduce job). On the webstatus page, there
are also 2 regions with an empty end key which seems very strange. One at
the end and one near to the middle. When I ran a mapreduce job over this
table, the region split startkey  is set however to the startkey of the next
region. (for the first region with an empty end key in the web interface)
(As a side node, stopping hbase took very long sometimes so I manually
killed the processes a few times before, which could have led to this...)

Shouldn't the regions be deleted when no data is present? (as I have set
versions to 1 and deleted the keys through HTable.deleteAll() function). I
manually run compactation through the webinterface without specifying any
key, but this didn't remove the regions (the compactations were executed on
the regionservers (as per log) and took 0 seconds to complete as there was
no data in them)


Also the startup phase is a lot longer than in hadoop 0.18.1. I have about
1500 regions over 7 servers, and it can take up to 5 minutes until all
regions are loaded. (Hbase doesn't even start to load regions, only when I
make a first request to it). But this could also be related to corrupt
regions?



Hbase settings:
 <property>
    <name>hbase.master.lease.period</name>
    <value>720000</value>
    <description>HMaster server lease period in milliseconds. Default is
    120 seconds.  Region servers must report in within this period else
    they are considered dead.  On loaded cluster, may need to up this
    period.</description>
  </property>
  
  <property>
    <name>hbase.regionserver.lease.period</name>
    <value>360000</value>
    <description>HRegion server lease period in milliseconds. Default is
    60 seconds. Clients must report in within this period else they are
    considered dead.</description>
  </property>
  
  
  <property>
    <name>hbase.hstore.compactionThreshold</name>
    <value>8</value>
    <description>
    If more than this number of HStoreFiles in any one HStore
    (one HStoreFile is written per flush of memcache) then a compaction
    is run to rewrite all HStoreFiles files as one.  Larger numbers
    put off compaction but when it runs, it takes longer to complete.
    During a compaction, updates cannot be flushed to disk.  Long
    compactions require memory sufficient to carry the logging of
    all updates across the duration of the compaction.
    
    If too large, clients timeout during compaction.
    </description>
  </property>
  <property>
    <name>hbase.hstore.compaction.max</name>
    <value>15</value>
    <description>Max number of HStoreFiles to compact per 'minor'
compaction.
    </description>
  </property>
  


Thanks for your help,
Thibaut
-- 
View this message in context: http://www.nabble.com/Table-with-80-regions-having-nearly-no-data-in-it-tp21058687p21058687.html
Sent from the HBase User mailing list archive at Nabble.com.


Re: Table with 80 regions having nearly no data in it

Posted by Thibaut_ <tb...@blue.lu>.
Hello St.Ack,

unfortunately I don't have the tables anymore (I did do a clean restart and
deleted all the tables).

If I encouter the missing region issue again, I will open up an issue. I
will also have DEBUG output available then.

As for the processes not stopping, I posted a stack dump and the the tail of
the log file in HBASE-1078.

Thibaut



stack-3 wrote:
> 
> Thibaut_ wrote:
>> Hello St.Ack,
>> thanks for your answer.
>>
>> I will vote on HBASE-880, that's exactly what I needed :)
>>
>> Except of the little glitch in the webinterface regarding the ending key,
>> that table seems to be ok then. I thought regions would get deleted. I'm
>> adding new data (Timestamp of current time is the key), and when I have
>> processed that data, I'm deleting the data. That's the reason why I get
>> more
>> and more regions... (because as soon as I add data that spans over two
>> regions, the first region won't be removed when I have deleted that
>> data).
>>   
> 
> We'd need a cluster cleaner process that would notice empty regions and 
> would merge them into adjacents.  That seems reasonable to me (Sounds 
> like HBASE-420).
> 
> The below is a bit hard to read but from what I can make out, its not 
> right.  The ENDKEY of a region should be the STARTKEY of the next.  The 
> ENDKEY of '', should be last in table.  Neither seems to be going on 
> here.   Perhaps make an issue and paste a clean output from .META. and 
> I'll take a look.
>> Here is the meta info of that region (NAME =>
>> 'tobeprocessed,12293840182411045696639,1229385024829') and the regions
>> around that region
>> .........
>>  tobeprocessed,1229383104274 column=info:regioninfo,
>> timestamp=1229384271404, value=REGION => {NAME => 'tobepr
>>  796785789,1229384269280    
>> ocessed,1229383104274796785789,1229384269280',
>> STARTKEY => '122938310427479678578
>>                              9', ENDKEY => '12293837601871695303679',
>> ENCODED => 1839260151, TABLE => {{NAME =
>>                              > 'tobeprocessed', IS_ROOT => 'false',
>> IS_META
>> => 'false', FAMILIES => [{NAME =>
>>                              'data', BLOOMFILTER => 'false', COMPRESSION
>> =>
>> 'NONE', VERSIONS => '1', LENGTH =>
>>                               '2147483647', TTL => '-1', IN_MEMORY =>
>> 'false', BLOCKCACHE => 'false'}], INDEXE
>>                              S => []}}
>>  tobeprocessed,1229383760187 column=info:regioninfo,
>>   
> 
> ....
>> But I have another table (rsssources), which when I scanned it yesterday
>> had
>>   
>>> 400 000 entries (count 'rsssources'), and today only has 180 000 entries
>>>     
>> (after killing hbase (kill -9), because it was unresponsive. 
> 
> Well, the above listing would seem to have missing regions.  Lets figure 
> whats gong on.  Make an issue 'missing regions' and paste in your scan 
> '.META.' output.  Do you think you can reproduce this?
> 
> 
>> I'm also
>> logging the DEBUG entries now to see what's happening at that point).
>>   
> 
> Good.  Can you see anything about the regions that go missing?
> 
>> When I execute a mapreduce job, a few regions don't seem to have any data
>> in
>> them. I did however never delete any data in that table (just replaces). 
>>
>> I did increase the timeout values, because I read somewhere else that it
>> would help in some cases. But I will reset the values to their original
>> values.
>>
>>
>> What's the best way to stop hbase when the hbase-stop script doesn't
>> work.
>> (Sometimes it just runs for hours... (probably a deadlock somewhere)?
>>
>>   
> Tail the master log with DEBUG enabled.  Should point at what is taking 
> so long to go down.  May indicate a particular regionserver.  Thread 
> dump it ("kill -QUIT PID").  Stick that in the issue to.
> 
> 
>> I'm waiting now for hbase to shut down, and will try to run the merge
>> script
>> on those two tables.
>>   
> Good stuff,
> St.Ack
> 
> 

-- 
View this message in context: http://www.nabble.com/Table-with-80-regions-having-nearly-no-data-in-it-tp21058687p21127237.html
Sent from the HBase User mailing list archive at Nabble.com.


Re: Table with 80 regions having nearly no data in it

Posted by stack <st...@duboce.net>.
Thibaut_ wrote:
> Hello St.Ack,
> thanks for your answer.
>
> I will vote on HBASE-880, that's exactly what I needed :)
>
> Except of the little glitch in the webinterface regarding the ending key,
> that table seems to be ok then. I thought regions would get deleted. I'm
> adding new data (Timestamp of current time is the key), and when I have
> processed that data, I'm deleting the data. That's the reason why I get more
> and more regions... (because as soon as I add data that spans over two
> regions, the first region won't be removed when I have deleted that data).
>   

We'd need a cluster cleaner process that would notice empty regions and 
would merge them into adjacents.  That seems reasonable to me (Sounds 
like HBASE-420).

The below is a bit hard to read but from what I can make out, its not 
right.  The ENDKEY of a region should be the STARTKEY of the next.  The 
ENDKEY of '', should be last in table.  Neither seems to be going on 
here.   Perhaps make an issue and paste a clean output from .META. and 
I'll take a look.
> Here is the meta info of that region (NAME =>
> 'tobeprocessed,12293840182411045696639,1229385024829') and the regions
> around that region
> .........
>  tobeprocessed,1229383104274 column=info:regioninfo,
> timestamp=1229384271404, value=REGION => {NAME => 'tobepr
>  796785789,1229384269280     ocessed,1229383104274796785789,1229384269280',
> STARTKEY => '122938310427479678578
>                              9', ENDKEY => '12293837601871695303679',
> ENCODED => 1839260151, TABLE => {{NAME =
>                              > 'tobeprocessed', IS_ROOT => 'false', IS_META
> => 'false', FAMILIES => [{NAME =>
>                              'data', BLOOMFILTER => 'false', COMPRESSION =>
> 'NONE', VERSIONS => '1', LENGTH =>
>                               '2147483647', TTL => '-1', IN_MEMORY =>
> 'false', BLOCKCACHE => 'false'}], INDEXE
>                              S => []}}
>  tobeprocessed,1229383760187 column=info:regioninfo,
>   

....
> But I have another table (rsssources), which when I scanned it yesterday had
>   
>> 400 000 entries (count 'rsssources'), and today only has 180 000 entries
>>     
> (after killing hbase (kill -9), because it was unresponsive. 

Well, the above listing would seem to have missing regions.  Lets figure 
whats gong on.  Make an issue 'missing regions' and paste in your scan 
'.META.' output.  Do you think you can reproduce this?


> I'm also
> logging the DEBUG entries now to see what's happening at that point).
>   

Good.  Can you see anything about the regions that go missing?

> When I execute a mapreduce job, a few regions don't seem to have any data in
> them. I did however never delete any data in that table (just replaces). 
>
> I did increase the timeout values, because I read somewhere else that it
> would help in some cases. But I will reset the values to their original
> values.
>
>
> What's the best way to stop hbase when the hbase-stop script doesn't work.
> (Sometimes it just runs for hours... (probably a deadlock somewhere)?
>
>   
Tail the master log with DEBUG enabled.  Should point at what is taking 
so long to go down.  May indicate a particular regionserver.  Thread 
dump it ("kill -QUIT PID").  Stick that in the issue to.


> I'm waiting now for hbase to shut down, and will try to run the merge script
> on those two tables.
>   
Good stuff,
St.Ack

Re: Table with 80 regions having nearly no data in it

Posted by Thibaut_ <tb...@blue.lu>.
Hello St.Ack,
thanks for your answer.

I will vote on HBASE-880, that's exactly what I needed :)

Except of the little glitch in the webinterface regarding the ending key,
that table seems to be ok then. I thought regions would get deleted. I'm
adding new data (Timestamp of current time is the key), and when I have
processed that data, I'm deleting the data. That's the reason why I get more
and more regions... (because as soon as I add data that spans over two
regions, the first region won't be removed when I have deleted that data).

Here is the meta info of that region (NAME =>
'tobeprocessed,12293840182411045696639,1229385024829') and the regions
around that region
.........
 tobeprocessed,1229383104274 column=info:regioninfo,
timestamp=1229384271404, value=REGION => {NAME => 'tobepr
 796785789,1229384269280     ocessed,1229383104274796785789,1229384269280',
STARTKEY => '122938310427479678578
                             9', ENDKEY => '12293837601871695303679',
ENCODED => 1839260151, TABLE => {{NAME =
                             > 'tobeprocessed', IS_ROOT => 'false', IS_META
=> 'false', FAMILIES => [{NAME =>
                             'data', BLOOMFILTER => 'false', COMPRESSION =>
'NONE', VERSIONS => '1', LENGTH =>
                              '2147483647', TTL => '-1', IN_MEMORY =>
'false', BLOCKCACHE => 'false'}], INDEXE
                             S => []}}
 tobeprocessed,1229383760187 column=info:regioninfo,
timestamp=1229385036201, value=REGION => {NAME => 'tobepr
 1695303679,1229385024829    ocessed,12293837601871695303679,1229385024829',
STARTKEY => '12293837601871695303
                             679', ENDKEY => '12293840182411045696639',
ENCODED => 1597707466, TABLE => {{NAME
                              => 'tobeprocessed', IS_ROOT => 'false',
IS_META => 'false', FAMILIES => [{NAME =
                             > 'data', BLOOMFILTER => 'false', COMPRESSION
=> 'NONE', VERSIONS => '1', LENGTH
                             => '2147483647', TTL => '-1', IN_MEMORY =>
'false', BLOCKCACHE => 'false'}], INDE
                             XES => []}}
 tobeprocessed,1229384018241 column=info:regioninfo,
timestamp=1229385036202, value=REGION => {NAME => 'tobepr
 1045696639,1229385024829    ocessed,12293840182411045696639,1229385024829',
STARTKEY => '12293840182411045696
                             639', ENDKEY => '', ENCODED => 2091963726,
TABLE => {{NAME => 'tobeprocessed', IS
                             _ROOT => 'false', IS_META => 'false', FAMILIES
=> [{NAME => 'data', BLOOMFILTER =
                             > 'false', COMPRESSION => 'NONE', VERSIONS =>
'1', LENGTH => '2147483647', TTL =>
                              '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}], INDEXES => []}}
 tobeprocessed,1229387389554 column=info:regioninfo,
timestamp=1229423191262, value=REGION => {NAME => 'tobepr
 1519264796,1229423166282    ocessed,12293873895541519264796,1229423166282',
STARTKEY => '12293873895541519264
                             796', ENDKEY => '12293964745931154000190',
ENCODED => 136070661, TABLE => {{NAME
                             => 'tobeprocessed', IS_ROOT => 'false', IS_META
=> 'false', FAMILIES => [{NAME =>
                              'data', BLOOMFILTER => 'false', COMPRESSION =>
'NONE', VERSIONS => '1', LENGTH =
                             > '2147483647', TTL => '-1', IN_MEMORY =>
'false', BLOCKCACHE => 'false'}], INDEX
                             ES => []}}
 tobeprocessed,1229396474593 column=info:regioninfo,
timestamp=1229427246552, value=REGION => {NAME => 'tobepr
 1154000190,1229427238214    ocessed,12293964745931154000190,1229427238214',
STARTKEY => '12293964745931154000
                             190', ENDKEY => '12294032229291208031240',
ENCODED => 1505006567, TABLE => {{NAME
                              => 'tobeprocessed', IS_ROOT => 'false',
IS_META => 'false', FAMILIES => [{NAME =
                             > 'data', BLOOMFILTER => 'false', COMPRESSION
=> 'NONE', VERSIONS => '1', LENGTH
                             => '2147483647', TTL => '-1', IN_MEMORY =>
'false', BLOCKCACHE => 'false'}], INDE
                             XES => []}}
............



But I have another table (rsssources), which when I scanned it yesterday had
> 400 000 entries (count 'rsssources'), and today only has 180 000 entries
(after killing hbase (kill -9), because it was unresponsive. I'm also
logging the DEBUG entries now to see what's happening at that point).
When I execute a mapreduce job, a few regions don't seem to have any data in
them. I did however never delete any data in that table (just replaces). 

I did increase the timeout values, because I read somewhere else that it
would help in some cases. But I will reset the values to their original
values.


What's the best way to stop hbase when the hbase-stop script doesn't work.
(Sometimes it just runs for hours... (probably a deadlock somewhere)?

I'm waiting now for hbase to shut down, and will try to run the merge script
on those two tables.

Thanks,
Thibaut



-- 
View this message in context: http://www.nabble.com/Table-with-80-regions-having-nearly-no-data-in-it-tp21058687p21074408.html
Sent from the HBase User mailing list archive at Nabble.com.


Re: Table with 80 regions having nearly no data in it

Posted by stack <st...@duboce.net>.
Thibaut_ wrote:
> As a side note, it would be helpfull if it would be possible not only to
> insert or change rows with BatchUpdate, but also to delete rows (So I can
> delete more rows at the end when I'm executing the other batched requests as
> well).
>
>   
HBASE-880.  Maybe add a vote?

> But something I have noticed is that I have a table (at least one)
>
> tobeprocessed 	{NAME => 'tobeprocessed', IS_ROOT => 'false', IS_META =>
> 'false', FAMILIES => [{NAME => 'data', BLOOMFILTER => 'false', COMPRESSION
> => 'NONE', VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY
> => 'false', BLOCKCACHE => 'false'}], INDEXES => []}
>
> which spans over 70 regions, but only has about 117 rows in it (just a few
> MByte). 

Did it only ever have 117 rows in it?  Or was it once many more than 
this and other rows were deleted?


> These entries are all in the last region (as I used a timestamp as
> key and I just checked with a mapreduce job). On the webstatus page, there
> are also 2 regions with an empty end key which seems very strange. 

Not 'strange' but 'broken'.  Where do you see that exactly?  Can you 
scan this table successfully?

Scan your .META. and paste in the info:regioninfo cell for each of these 
regions so we can take a look.

> One at
> the end and one near to the middle. When I ran a mapreduce job over this
> table, the region split startkey  is set however to the startkey of the next
> region. (for the first region with an empty end key in the web interface)
> (As a side node, stopping hbase took very long sometimes so I manually
> killed the processes a few times before, which could have led to this...)
>
>   
OK.  It might have damaged it though 0.19.0 should be more resilient 
than past versions.


> Shouldn't the regions be deleted when no data is present? (as I have set
> versions to 1 and deleted the keys through HTable.deleteAll() function). 

Not currently.  Once made a region remains though after deletes it has 
no data.

You could merge up all of these empty regions but you'd have to shutdown 
hbase and run the merge tool (We should add it to the new UI as an 
option under the new manual split/force-compaction feature).

> Also the startup phase is a lot longer than in hadoop 0.18.1. I have about
> 1500 regions over 7 servers, and it can take up to 5 minutes until all
> regions are loaded. (Hbase doesn't even start to load regions, only when I
> make a first request to it). But this could also be related to corrupt
> regions?
>
>   
This is being looked into JBA.  Hopefully we can improve here before the 
release.  Study your master log with DEBUG enabled.  Whats it up too?  
There is a new 'safe mode' in hbase.  Maybe this goes on too long.  Is 
it assigning regions?  Are they taking a long time to open?


>
> Hbase settings:
>  <property>
>     <name>hbase.master.lease.period</name>
>     <value>720000</value>
>     <description>HMaster server lease period in milliseconds. Default is
>     120 seconds.  Region servers must report in within this period else
>     they are considered dead.  On loaded cluster, may need to up this
>     period.</description>
>   </property>
>   

Did you find that you needed it to be this long?
St.Ack