You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vidhyashankar Venkataraman <vi...@yahoo-inc.com> on 2010/08/26 20:59:10 UTC
Region splits in 0.89...
My hbase table issued a mass split after I loaded regions with greater sizes than maxfilesize.. (my bad..)
Now, when I try accessing the master through the web interface, it just hangs...
And, if I scan the META, I get the parent regions set to offline.. And the child regions have random byte keys (that was part of the HBASE 2515).. But one thing that troubles me is that even the split child regions don't seem to have been assigned to any server according to META: I am not sure if this is expected..
DocDB,0000000003617973,12 column=info:regioninfo, timestamp=1282810230593, value=REGION => {NAME =>
82331439883.6a66354e2c587 'DocDB,0000000003617973,1282331439883.6a66354e2c58779278942a8e5cf152ca.'
79278942a8e5cf152ca. , STARTKEY => '0000000003617973', ENDKEY => '0000000003751972', ENCODED =
> 6a66354e2c58779278942a8e5cf152ca, OFFLINE => true, SPLIT => true, TABLE
=> {{NAME => 'DocDB', MAX_FILESIZE => '4294967296', FAMILIES => [{NAME =
> 'bigColumn', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS
=> '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '1048576
', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
DocDB,0000000003617973,12 column=info:server, timestamp=1282810230593, value=
82331439883.6a66354e2c587
79278942a8e5cf152ca.
DocDB,0000000003617973,12 column=info:serverstartcode, timestamp=1282810230593, value=
82331439883.6a66354e2c587
79278942a8e5cf152ca.
DocDB,0000000003617973,12 column=info:splitA, timestamp=1282810230593, value=\x00\x1000000000036849
82331439883.6a66354e2c587 63\x00\x00\x00\x01*\xADr\xB4\x82FDocDB,0000000003617973,1282810229890.57f
79278942a8e5cf152ca. 2eed2d43cece270244a39233f5a5a.\x00\x100000000003617973\x00\x00\x00\x05\x0
5DocDB\x00\x00\x00\x00\x00\x03\x00\x00\x00\x07IS_ROOT\x00\x00\x00\x05fals
e\x00\x00\x00\x07IS_META\x00\x00\x00\x05false\x00\x00\x00\x0CMAX_FILESIZE
\x00\x00\x00\x0A4294967296\x00\x00\x00\x01\x08\x09bigColumn\x00\x00\x00\x
08\x00\x00\x00\x0BBLOOMFILTER\x00\x00\x00\x04NONE\x00\x00\x00\x11REPLICAT
ION_SCOPE\x00\x00\x00\x010\x00\x00\x00\x0BCOMPRESSION\x00\x00\x00\x04NONE
\x00\x00\x00\x08VERSIONS\x00\x00\x00\x011\x00\x00\x00\x03TTL\x00\x00\x00\
x0A2147483647\x00\x00\x00\x09BLOCKSIZE\x00\x00\x00\x071048576\x00\x00\x00
\x09IN_MEMORY\x00\x00\x00\x05false\x00\x00\x00\x0ABLOCKCACHE\x00\x00\x00\
x05false0\xF1\xD4\xBB
DocDB,0000000003617973,12 column=info:splitA_checked, timestamp=1282815236171, value=\x01
82331439883.6a66354e2c587
79278942a8e5cf152ca.
DocDB,0000000003617973,12 column=info:splitB, timestamp=1282810230593, value=\x00\x1000000000037519
82331439883.6a66354e2c587 72\x00\x00\x00\x01*\xADr\xB4\x82FDocDB,0000000003684963,1282810229890.5fe
79278942a8e5cf152ca. 1f7c4de1640642fb8454a1b8c623c.\x00\x100000000003684963\x00\x00\x00\x05\x0
5DocDB\x00\x00\x00\x00\x00\x03\x00\x00\x00\x07IS_ROOT\x00\x00\x00\x05fals
e\x00\x00\x00\x07IS_META\x00\x00\x00\x05false\x00\x00\x00\x0CMAX_FILESIZE
\x00\x00\x00\x0A4294967296\x00\x00\x00\x01\x08\x09bigColumn\x00\x00\x00\x
08\x00\x00\x00\x0BBLOOMFILTER\x00\x00\x00\x04NONE\x00\x00\x00\x11REPLICAT
ION_SCOPE\x00\x00\x00\x010\x00\x00\x00\x0BCOMPRESSION\x00\x00\x00\x04NONE
\x00\x00\x00\x08VERSIONS\x00\x00\x00\x011\x00\x00\x00\x03TTL\x00\x00\x00\
x0A2147483647\x00\x00\x00\x09BLOCKSIZE\x00\x00\x00\x071048576\x00\x00\x00
\x09IN_MEMORY\x00\x00\x00\x05false\x00\x00\x00\x0ABLOCKCACHE\x00\x00\x00\
x05false\xB54N\x1E
DocDB,0000000003617973,12 column=info:splitB_checked, timestamp=1282815236219, value=\x01
82331439883.6a66354e2c587
79278942a8e5cf152ca.
Re: Region splits in 0.89...
Posted by Vidhyashankar Venkataraman <vi...@yahoo-inc.com>.
>> It's possible that the master is slowly assigning out all the daughter
>> regions, and it's just taking a really long time since you loaded so many.
I forgot to add it the last time.. But I think that's what is happening.. Master getting choked up because of a flash crowd of splits..
(Anyways, I just wanted to verify if the META table scan was a possible output)...
I will reload the data with some altered configs..
Thank you
Vidhya
On 8/26/10 1:16 PM, "Todd Lipcon" <to...@cloudera.com> wrote:
Hey Vidhya,
Anything interesting in the logs on the master?
It's possible that the master is slowly assigning out all the daughter
regions, and it's just taking a really long time since you loaded so many.
-Todd
On Thu, Aug 26, 2010 at 11:59 AM, Vidhyashankar Venkataraman <
vidhyash@yahoo-inc.com> wrote:
> My hbase table issued a mass split after I loaded regions with greater
> sizes than maxfilesize.. (my bad..)
>
> Now, when I try accessing the master through the web interface, it just
> hangs...
>
> And, if I scan the META, I get the parent regions set to offline.. And the
> child regions have random byte keys (that was part of the HBASE 2515).. But
> one thing that troubles me is that even the split child regions don't seem
> to have been assigned to any server according to META: I am not sure if
> this is expected..
>
>
> DocDB,0000000003617973,12 column=info:regioninfo,
> timestamp=1282810230593, value=REGION => {NAME =>
> 82331439883.6a66354e2c587
> 'DocDB,0000000003617973,1282331439883.6a66354e2c58779278942a8e5cf152ca.'
> 79278942a8e5cf152ca. , STARTKEY => '0000000003617973',
> ENDKEY => '0000000003751972', ENCODED =
> >
> 6a66354e2c58779278942a8e5cf152ca, OFFLINE => true, SPLIT => true, TABLE
> => {{NAME => 'DocDB', MAX_FILESIZE =>
> '4294967296', FAMILIES => [{NAME =
> > 'bigColumn', BLOOMFILTER => 'NONE',
> REPLICATION_SCOPE => '0', VERSIONS
> => '1', COMPRESSION => 'NONE', TTL =>
> '2147483647', BLOCKSIZE => '1048576
> ', IN_MEMORY => 'false', BLOCKCACHE =>
> 'false'}]}}
> DocDB,0000000003617973,12 column=info:server,
> timestamp=1282810230593, value=
> 82331439883.6a66354e2c587
> 79278942a8e5cf152ca.
> DocDB,0000000003617973,12 column=info:serverstartcode,
> timestamp=1282810230593, value=
> 82331439883.6a66354e2c587
> 79278942a8e5cf152ca.
> DocDB,0000000003617973,12 column=info:splitA,
> timestamp=1282810230593, value=\x00\x1000000000036849
> 82331439883.6a66354e2c587
> 63\x00\x00\x00\x01*\xADr\xB4\x82FDocDB,0000000003617973,1282810229890.57f
> 79278942a8e5cf152ca.
> 2eed2d43cece270244a39233f5a5a.\x00\x100000000003617973\x00\x00\x00\x05\x0
>
> 5DocDB\x00\x00\x00\x00\x00\x03\x00\x00\x00\x07IS_ROOT\x00\x00\x00\x05fals
>
> e\x00\x00\x00\x07IS_META\x00\x00\x00\x05false\x00\x00\x00\x0CMAX_FILESIZE
>
> \x00\x00\x00\x0A4294967296\x00\x00\x00\x01\x08\x09bigColumn\x00\x00\x00\x
>
> 08\x00\x00\x00\x0BBLOOMFILTER\x00\x00\x00\x04NONE\x00\x00\x00\x11REPLICAT
>
> ION_SCOPE\x00\x00\x00\x010\x00\x00\x00\x0BCOMPRESSION\x00\x00\x00\x04NONE
>
> \x00\x00\x00\x08VERSIONS\x00\x00\x00\x011\x00\x00\x00\x03TTL\x00\x00\x00\
>
> x0A2147483647\x00\x00\x00\x09BLOCKSIZE\x00\x00\x00\x071048576\x00\x00\x00
>
> \x09IN_MEMORY\x00\x00\x00\x05false\x00\x00\x00\x0ABLOCKCACHE\x00\x00\x00\
> x05false0\xF1\xD4\xBB
> DocDB,0000000003617973,12 column=info:splitA_checked,
> timestamp=1282815236171, value=\x01
> 82331439883.6a66354e2c587
> 79278942a8e5cf152ca.
> DocDB,0000000003617973,12 column=info:splitB,
> timestamp=1282810230593, value=\x00\x1000000000037519
> 82331439883.6a66354e2c587
> 72\x00\x00\x00\x01*\xADr\xB4\x82FDocDB,0000000003684963,1282810229890.5fe
> 79278942a8e5cf152ca.
> 1f7c4de1640642fb8454a1b8c623c.\x00\x100000000003684963\x00\x00\x00\x05\x0
>
> 5DocDB\x00\x00\x00\x00\x00\x03\x00\x00\x00\x07IS_ROOT\x00\x00\x00\x05fals
>
> e\x00\x00\x00\x07IS_META\x00\x00\x00\x05false\x00\x00\x00\x0CMAX_FILESIZE
>
> \x00\x00\x00\x0A4294967296\x00\x00\x00\x01\x08\x09bigColumn\x00\x00\x00\x
>
> 08\x00\x00\x00\x0BBLOOMFILTER\x00\x00\x00\x04NONE\x00\x00\x00\x11REPLICAT
>
> ION_SCOPE\x00\x00\x00\x010\x00\x00\x00\x0BCOMPRESSION\x00\x00\x00\x04NONE
>
> \x00\x00\x00\x08VERSIONS\x00\x00\x00\x011\x00\x00\x00\x03TTL\x00\x00\x00\
>
> x0A2147483647\x00\x00\x00\x09BLOCKSIZE\x00\x00\x00\x071048576\x00\x00\x00
>
> \x09IN_MEMORY\x00\x00\x00\x05false\x00\x00\x00\x0ABLOCKCACHE\x00\x00\x00\
> x05false\xB54N\x1E
> DocDB,0000000003617973,12 column=info:splitB_checked,
> timestamp=1282815236219, value=\x01
> 82331439883.6a66354e2c587
> 79278942a8e5cf152ca.
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Region splits in 0.89...
Posted by Todd Lipcon <to...@cloudera.com>.
Hey Vidhya,
Anything interesting in the logs on the master?
It's possible that the master is slowly assigning out all the daughter
regions, and it's just taking a really long time since you loaded so many.
-Todd
On Thu, Aug 26, 2010 at 11:59 AM, Vidhyashankar Venkataraman <
vidhyash@yahoo-inc.com> wrote:
> My hbase table issued a mass split after I loaded regions with greater
> sizes than maxfilesize.. (my bad..)
>
> Now, when I try accessing the master through the web interface, it just
> hangs...
>
> And, if I scan the META, I get the parent regions set to offline.. And the
> child regions have random byte keys (that was part of the HBASE 2515).. But
> one thing that troubles me is that even the split child regions don't seem
> to have been assigned to any server according to META: I am not sure if
> this is expected..
>
>
> DocDB,0000000003617973,12 column=info:regioninfo,
> timestamp=1282810230593, value=REGION => {NAME =>
> 82331439883.6a66354e2c587
> 'DocDB,0000000003617973,1282331439883.6a66354e2c58779278942a8e5cf152ca.'
> 79278942a8e5cf152ca. , STARTKEY => '0000000003617973',
> ENDKEY => '0000000003751972', ENCODED =
> >
> 6a66354e2c58779278942a8e5cf152ca, OFFLINE => true, SPLIT => true, TABLE
> => {{NAME => 'DocDB', MAX_FILESIZE =>
> '4294967296', FAMILIES => [{NAME =
> > 'bigColumn', BLOOMFILTER => 'NONE',
> REPLICATION_SCOPE => '0', VERSIONS
> => '1', COMPRESSION => 'NONE', TTL =>
> '2147483647', BLOCKSIZE => '1048576
> ', IN_MEMORY => 'false', BLOCKCACHE =>
> 'false'}]}}
> DocDB,0000000003617973,12 column=info:server,
> timestamp=1282810230593, value=
> 82331439883.6a66354e2c587
> 79278942a8e5cf152ca.
> DocDB,0000000003617973,12 column=info:serverstartcode,
> timestamp=1282810230593, value=
> 82331439883.6a66354e2c587
> 79278942a8e5cf152ca.
> DocDB,0000000003617973,12 column=info:splitA,
> timestamp=1282810230593, value=\x00\x1000000000036849
> 82331439883.6a66354e2c587
> 63\x00\x00\x00\x01*\xADr\xB4\x82FDocDB,0000000003617973,1282810229890.57f
> 79278942a8e5cf152ca.
> 2eed2d43cece270244a39233f5a5a.\x00\x100000000003617973\x00\x00\x00\x05\x0
>
> 5DocDB\x00\x00\x00\x00\x00\x03\x00\x00\x00\x07IS_ROOT\x00\x00\x00\x05fals
>
> e\x00\x00\x00\x07IS_META\x00\x00\x00\x05false\x00\x00\x00\x0CMAX_FILESIZE
>
> \x00\x00\x00\x0A4294967296\x00\x00\x00\x01\x08\x09bigColumn\x00\x00\x00\x
>
> 08\x00\x00\x00\x0BBLOOMFILTER\x00\x00\x00\x04NONE\x00\x00\x00\x11REPLICAT
>
> ION_SCOPE\x00\x00\x00\x010\x00\x00\x00\x0BCOMPRESSION\x00\x00\x00\x04NONE
>
> \x00\x00\x00\x08VERSIONS\x00\x00\x00\x011\x00\x00\x00\x03TTL\x00\x00\x00\
>
> x0A2147483647\x00\x00\x00\x09BLOCKSIZE\x00\x00\x00\x071048576\x00\x00\x00
>
> \x09IN_MEMORY\x00\x00\x00\x05false\x00\x00\x00\x0ABLOCKCACHE\x00\x00\x00\
> x05false0\xF1\xD4\xBB
> DocDB,0000000003617973,12 column=info:splitA_checked,
> timestamp=1282815236171, value=\x01
> 82331439883.6a66354e2c587
> 79278942a8e5cf152ca.
> DocDB,0000000003617973,12 column=info:splitB,
> timestamp=1282810230593, value=\x00\x1000000000037519
> 82331439883.6a66354e2c587
> 72\x00\x00\x00\x01*\xADr\xB4\x82FDocDB,0000000003684963,1282810229890.5fe
> 79278942a8e5cf152ca.
> 1f7c4de1640642fb8454a1b8c623c.\x00\x100000000003684963\x00\x00\x00\x05\x0
>
> 5DocDB\x00\x00\x00\x00\x00\x03\x00\x00\x00\x07IS_ROOT\x00\x00\x00\x05fals
>
> e\x00\x00\x00\x07IS_META\x00\x00\x00\x05false\x00\x00\x00\x0CMAX_FILESIZE
>
> \x00\x00\x00\x0A4294967296\x00\x00\x00\x01\x08\x09bigColumn\x00\x00\x00\x
>
> 08\x00\x00\x00\x0BBLOOMFILTER\x00\x00\x00\x04NONE\x00\x00\x00\x11REPLICAT
>
> ION_SCOPE\x00\x00\x00\x010\x00\x00\x00\x0BCOMPRESSION\x00\x00\x00\x04NONE
>
> \x00\x00\x00\x08VERSIONS\x00\x00\x00\x011\x00\x00\x00\x03TTL\x00\x00\x00\
>
> x0A2147483647\x00\x00\x00\x09BLOCKSIZE\x00\x00\x00\x071048576\x00\x00\x00
>
> \x09IN_MEMORY\x00\x00\x00\x05false\x00\x00\x00\x0ABLOCKCACHE\x00\x00\x00\
> x05false\xB54N\x1E
> DocDB,0000000003617973,12 column=info:splitB_checked,
> timestamp=1282815236219, value=\x01
> 82331439883.6a66354e2c587
> 79278942a8e5cf152ca.
>
--
Todd Lipcon
Software Engineer, Cloudera