You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Igor Ranitovic <ir...@gmail.com> on 2009/09/02 21:57:29 UTC
major_compaction and LZO
After setting up a test table with about 250K rows, I wanted to enable
lzo compression on it.
In the shell, I disabled and altered the table and then ran major_compaction.
This is the result:
hbase(main):013: major_compact 'urls'
0 row(s) in 0.0460 seconds
Does '0 rows' means that this did not work?
(the size of the table on the disk did not change/reduce)
Table info:
hbase(main):019:0> describe 'urls'
DESCRIPTION
ENABLED
false
{NAME => 'urls', FAMILIES => [{NAME => 'data', VERSIONS => '1',
COMPRE
SSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
=
> 'false', BLOCKCACHE => 'true'}]}
1 row(s) in 0.0420 seconds
Table Count:
hbase(main):021:0> count 'urls'
....
Current count: 251000, row: http://com,zz/
>From the master log:
hbase-crawler-master-xxx.log:2009-09-02 13:54:51,880 INFO
com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl
library
hbase-crawler-master-xxx.log:2009-09-02 13:54:51,891 INFO
com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized
native-lzo library
I using 0.20. Thanks.
i.
Re: major_compaction and LZO
Posted by Igor Ranitovic <ir...@gmail.com>.
The compression eventually happened. I am not sure when, but it seems it
happened after adding some more data to the table.
Thank you all.
i.
Andrew Purtell wrote:
> libgplcompression.so in hbase/lib/native/<arch> on all region server nodes?
>
> hadoop-gpl-compression.jar in hbase/lib on all region server nodes?
>
> Run at DEBUG log level and initiate major_compaction. Anything relevant in the master log? Anything relevant in the region server logs?
>
> - Andy
>
>
>
>
> ________________________________
> From: Igor Ranitovic <ir...@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Wednesday, September 2, 2009 2:52:11 PM
> Subject: Re: major_compaction and LZO
>
> I checked a HFile and it does not look compressed.
> Any ideas on how to proceed with troubleshooting?
> Thanks.
>
> i.
>
>> Regionserver logs will mention if major compaction happened on a region
>> or not, but maybe only at DEBUG log level, I'm not sure. I was running
>> with DEBUG at the time and saw immediately on a region server log tail
>> that the store files were rewritten with LZO.
>>
>> I downloaded a HFile out of a randomly chosen region via 'hadoop fs -copyToLocal ...' and looked at it to confirm it was compressed.
>>
>>> hbase(main):013: major_compact 'urls'
>>> 0 row(s) in 0.0460 seconds
>> 0 row result for an admin command doesn't mean anything. This tells you the command was accepted. If there was a problem, an exception would be thrown.
>>
>> - Andy
>>
>>
>>
>>
>> ________________________________
>> From: Igor Ranitovic <ir...@gmail.com>
>> To: hbase-user@hadoop.apache.org
>> Sent: Wednesday, September 2, 2009 2:12:37 PM
>> Subject: Re: major_compaction and LZO
>>
>> I tried again, but there is nothing in the regionserver logs.
>> The size of the table is tiny (MBs).
>>
>> In general, what would be a good way to verify if LZO compression is
>> working.
>>
>> i.
>>
>>> major compaction is not instant... I've seen it take an hour on a
>>> 300-400gb table. You want to check the regionserver logs looking for
>>> any major compaction lines.
>>>
>>> in your case, the table will take a lot less longer, but still not
>>> instant. Give it another shot, see what happens.
>>>
>>> -ryan
>>>
>>> On Wed, Sep 2, 2009 at 12:57 PM, Igor Ranitovic<ir...@gmail.com> wrote:
>>>> After setting up a test table with about 250K rows, I wanted to enable
>>>> lzo compression on it.
>>>>
>>>> In the shell, I disabled and altered the table and then ran major_compaction.
>>>>
>>>> This is the result:
>>>>
>>>> hbase(main):013: major_compact 'urls'
>>>> 0 row(s) in 0.0460 seconds
>>>>
>>>> Does '0 rows' means that this did not work?
>>>> (the size of the table on the disk did not change/reduce)
>>>>
>>>> Table info:
>>>> hbase(main):019:0> describe 'urls'
>>>> DESCRIPTION
>>>> ENABLED
>>>>
>>>> false
>>>> {NAME => 'urls', FAMILIES => [{NAME => 'data', VERSIONS => '1',
>>>> COMPRE
>>>> SSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
>>>> =
>>>> > 'false', BLOCKCACHE => 'true'}]}
>>>> 1 row(s) in 0.0420 seconds
>>>>
>>>>
>>>> Table Count:
>>>> hbase(main):021:0> count 'urls'
>>>> ....
>>>> Current count: 251000, row: http://com,zz/
>>>>
>>>>
>>>> From the master log:
>>>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,880 INFO
>>>> com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl
>>>> library
>>>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,891 INFO
>>>> com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized
>>>> native-lzo library
>>>>
>>>> I using 0.20. Thanks.
>>>>
>>>> i.
>>>>
>>
>>
>
>
>
Re: major_compaction and LZO
Posted by Andrew Purtell <ap...@apache.org>.
libgplcompression.so in hbase/lib/native/<arch> on all region server nodes?
hadoop-gpl-compression.jar in hbase/lib on all region server nodes?
Run at DEBUG log level and initiate major_compaction. Anything relevant in the master log? Anything relevant in the region server logs?
- Andy
________________________________
From: Igor Ranitovic <ir...@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Wednesday, September 2, 2009 2:52:11 PM
Subject: Re: major_compaction and LZO
I checked a HFile and it does not look compressed.
Any ideas on how to proceed with troubleshooting?
Thanks.
i.
> Regionserver logs will mention if major compaction happened on a region
> or not, but maybe only at DEBUG log level, I'm not sure. I was running
> with DEBUG at the time and saw immediately on a region server log tail
> that the store files were rewritten with LZO.
>
> I downloaded a HFile out of a randomly chosen region via 'hadoop fs -copyToLocal ...' and looked at it to confirm it was compressed.
>
>> hbase(main):013: major_compact 'urls'
>> 0 row(s) in 0.0460 seconds
>
> 0 row result for an admin command doesn't mean anything. This tells you the command was accepted. If there was a problem, an exception would be thrown.
>
> - Andy
>
>
>
>
> ________________________________
> From: Igor Ranitovic <ir...@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Wednesday, September 2, 2009 2:12:37 PM
> Subject: Re: major_compaction and LZO
>
> I tried again, but there is nothing in the regionserver logs.
> The size of the table is tiny (MBs).
>
> In general, what would be a good way to verify if LZO compression is
> working.
>
> i.
>
>> major compaction is not instant... I've seen it take an hour on a
>> 300-400gb table. You want to check the regionserver logs looking for
>> any major compaction lines.
>>
>> in your case, the table will take a lot less longer, but still not
>> instant. Give it another shot, see what happens.
>>
>> -ryan
>>
>> On Wed, Sep 2, 2009 at 12:57 PM, Igor Ranitovic<ir...@gmail.com> wrote:
>>> After setting up a test table with about 250K rows, I wanted to enable
>>> lzo compression on it.
>>>
>>> In the shell, I disabled and altered the table and then ran major_compaction.
>>>
>>> This is the result:
>>>
>>> hbase(main):013: major_compact 'urls'
>>> 0 row(s) in 0.0460 seconds
>>>
>>> Does '0 rows' means that this did not work?
>>> (the size of the table on the disk did not change/reduce)
>>>
>>> Table info:
>>> hbase(main):019:0> describe 'urls'
>>> DESCRIPTION
>>> ENABLED
>>>
>>> false
>>> {NAME => 'urls', FAMILIES => [{NAME => 'data', VERSIONS => '1',
>>> COMPRE
>>> SSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
>>> =
>>> > 'false', BLOCKCACHE => 'true'}]}
>>> 1 row(s) in 0.0420 seconds
>>>
>>>
>>> Table Count:
>>> hbase(main):021:0> count 'urls'
>>> ....
>>> Current count: 251000, row: http://com,zz/
>>>
>>>
>>> From the master log:
>>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,880 INFO
>>> com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl
>>> library
>>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,891 INFO
>>> com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized
>>> native-lzo library
>>>
>>> I using 0.20. Thanks.
>>>
>>> i.
>>>
>
>
>
Re: major_compaction and LZO
Posted by Ryan Rawson <ry...@gmail.com>.
Use the tool in hfile, it shows you what codec the file was written with.
On Sep 2, 2009 2:52 PM, "Igor Ranitovic" <ir...@gmail.com> wrote:
I checked a HFile and it does not look compressed.
Any ideas on how to proceed with troubleshooting?
Thanks.
i.
> Regionserver logs will mention if major compaction happened on a region >
or not, but maybe only...
Re: major_compaction and LZO
Posted by Igor Ranitovic <ir...@gmail.com>.
I checked a HFile and it does not look compressed.
Any ideas on how to proceed with troubleshooting?
Thanks.
i.
> Regionserver logs will mention if major compaction happened on a region
> or not, but maybe only at DEBUG log level, I'm not sure. I was running
> with DEBUG at the time and saw immediately on a region server log tail
> that the store files were rewritten with LZO.
>
> I downloaded a HFile out of a randomly chosen region via 'hadoop fs -copyToLocal ...' and looked at it to confirm it was compressed.
>
>> hbase(main):013: major_compact 'urls'
>> 0 row(s) in 0.0460 seconds
>
> 0 row result for an admin command doesn't mean anything. This tells you the command was accepted. If there was a problem, an exception would be thrown.
>
> - Andy
>
>
>
>
> ________________________________
> From: Igor Ranitovic <ir...@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Wednesday, September 2, 2009 2:12:37 PM
> Subject: Re: major_compaction and LZO
>
> I tried again, but there is nothing in the regionserver logs.
> The size of the table is tiny (MBs).
>
> In general, what would be a good way to verify if LZO compression is
> working.
>
> i.
>
>> major compaction is not instant... I've seen it take an hour on a
>> 300-400gb table. You want to check the regionserver logs looking for
>> any major compaction lines.
>>
>> in your case, the table will take a lot less longer, but still not
>> instant. Give it another shot, see what happens.
>>
>> -ryan
>>
>> On Wed, Sep 2, 2009 at 12:57 PM, Igor Ranitovic<ir...@gmail.com> wrote:
>>> After setting up a test table with about 250K rows, I wanted to enable
>>> lzo compression on it.
>>>
>>> In the shell, I disabled and altered the table and then ran major_compaction.
>>>
>>> This is the result:
>>>
>>> hbase(main):013: major_compact 'urls'
>>> 0 row(s) in 0.0460 seconds
>>>
>>> Does '0 rows' means that this did not work?
>>> (the size of the table on the disk did not change/reduce)
>>>
>>> Table info:
>>> hbase(main):019:0> describe 'urls'
>>> DESCRIPTION
>>> ENABLED
>>>
>>> false
>>> {NAME => 'urls', FAMILIES => [{NAME => 'data', VERSIONS => '1',
>>> COMPRE
>>> SSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
>>> =
>>> > 'false', BLOCKCACHE => 'true'}]}
>>> 1 row(s) in 0.0420 seconds
>>>
>>>
>>> Table Count:
>>> hbase(main):021:0> count 'urls'
>>> ....
>>> Current count: 251000, row: http://com,zz/
>>>
>>>
>>> From the master log:
>>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,880 INFO
>>> com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl
>>> library
>>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,891 INFO
>>> com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized
>>> native-lzo library
>>>
>>> I using 0.20. Thanks.
>>>
>>> i.
>>>
>
>
>
Re: major_compaction and LZO
Posted by Ryan Rawson <ry...@gmail.com>.
The main in the HFile class dumps stats and other things too. No downloading
necessary either.
On Sep 2, 2009 2:30 PM, "Andrew Purtell" <ap...@apache.org> wrote:
Regionserver logs will mention if major compaction happened on a region
or not, but maybe only at DEBUG log level, I'm not sure. I was running
with DEBUG at the time and saw immediately on a region server log tail
that the store files were rewritten with LZO.
I downloaded a HFile out of a randomly chosen region via 'hadoop fs
-copyToLocal ...' and looked at it to confirm it was compressed.
> hbase(main):013: major_compact 'urls' > 0 row(s) in 0.0460 seconds
0 row result for an admin command doesn't mean anything. This tells you the
command was accepted. If there was a problem, an exception would be thrown.
- Andy
________________________________
From: Igor Ranitovic <ir...@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Wednesday, September 2, 2009 2:12:37 PM
Subject: Re: major_compaction and LZO
I tried again, but there is nothing in the regionserver logs. The size of
the table is tiny (MBs). ...
Re: major_compaction and LZO
Posted by Andrew Purtell <ap...@apache.org>.
Regionserver logs will mention if major compaction happened on a region
or not, but maybe only at DEBUG log level, I'm not sure. I was running
with DEBUG at the time and saw immediately on a region server log tail
that the store files were rewritten with LZO.
I downloaded a HFile out of a randomly chosen region via 'hadoop fs -copyToLocal ...' and looked at it to confirm it was compressed.
> hbase(main):013: major_compact 'urls'
> 0 row(s) in 0.0460 seconds
0 row result for an admin command doesn't mean anything. This tells you the command was accepted. If there was a problem, an exception would be thrown.
- Andy
________________________________
From: Igor Ranitovic <ir...@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Wednesday, September 2, 2009 2:12:37 PM
Subject: Re: major_compaction and LZO
I tried again, but there is nothing in the regionserver logs.
The size of the table is tiny (MBs).
In general, what would be a good way to verify if LZO compression is
working.
i.
> major compaction is not instant... I've seen it take an hour on a
> 300-400gb table. You want to check the regionserver logs looking for
> any major compaction lines.
>
> in your case, the table will take a lot less longer, but still not
> instant. Give it another shot, see what happens.
>
> -ryan
>
> On Wed, Sep 2, 2009 at 12:57 PM, Igor Ranitovic<ir...@gmail.com> wrote:
>> After setting up a test table with about 250K rows, I wanted to enable
>> lzo compression on it.
>>
>> In the shell, I disabled and altered the table and then ran major_compaction.
>>
>> This is the result:
>>
>> hbase(main):013: major_compact 'urls'
>> 0 row(s) in 0.0460 seconds
>>
>> Does '0 rows' means that this did not work?
>> (the size of the table on the disk did not change/reduce)
>>
>> Table info:
>> hbase(main):019:0> describe 'urls'
>> DESCRIPTION
>> ENABLED
>>
>> false
>> {NAME => 'urls', FAMILIES => [{NAME => 'data', VERSIONS => '1',
>> COMPRE
>> SSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
>> =
>> > 'false', BLOCKCACHE => 'true'}]}
>> 1 row(s) in 0.0420 seconds
>>
>>
>> Table Count:
>> hbase(main):021:0> count 'urls'
>> ....
>> Current count: 251000, row: http://com,zz/
>>
>>
>> From the master log:
>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,880 INFO
>> com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl
>> library
>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,891 INFO
>> com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized
>> native-lzo library
>>
>> I using 0.20. Thanks.
>>
>> i.
>>
>
Re: major_compaction and LZO
Posted by Igor Ranitovic <ir...@gmail.com>.
I tried again, but there is nothing in the regionserver logs.
The size of the table is tiny (MBs).
In general, what would be a good way to verify if LZO compression is
working.
i.
> major compaction is not instant... I've seen it take an hour on a
> 300-400gb table. You want to check the regionserver logs looking for
> any major compaction lines.
>
> in your case, the table will take a lot less longer, but still not
> instant. Give it another shot, see what happens.
>
> -ryan
>
> On Wed, Sep 2, 2009 at 12:57 PM, Igor Ranitovic<ir...@gmail.com> wrote:
>> After setting up a test table with about 250K rows, I wanted to enable
>> lzo compression on it.
>>
>> In the shell, I disabled and altered the table and then ran major_compaction.
>>
>> This is the result:
>>
>> hbase(main):013: major_compact 'urls'
>> 0 row(s) in 0.0460 seconds
>>
>> Does '0 rows' means that this did not work?
>> (the size of the table on the disk did not change/reduce)
>>
>> Table info:
>> hbase(main):019:0> describe 'urls'
>> DESCRIPTION
>> ENABLED
>>
>> false
>> {NAME => 'urls', FAMILIES => [{NAME => 'data', VERSIONS => '1',
>> COMPRE
>> SSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
>> =
>> > 'false', BLOCKCACHE => 'true'}]}
>> 1 row(s) in 0.0420 seconds
>>
>>
>> Table Count:
>> hbase(main):021:0> count 'urls'
>> ....
>> Current count: 251000, row: http://com,zz/
>>
>>
>> From the master log:
>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,880 INFO
>> com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl
>> library
>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,891 INFO
>> com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized
>> native-lzo library
>>
>> I using 0.20. Thanks.
>>
>> i.
>>
>
Re: major_compaction and LZO
Posted by Ryan Rawson <ry...@gmail.com>.
major compaction is not instant... I've seen it take an hour on a
300-400gb table. You want to check the regionserver logs looking for
any major compaction lines.
in your case, the table will take a lot less longer, but still not
instant. Give it another shot, see what happens.
-ryan
On Wed, Sep 2, 2009 at 12:57 PM, Igor Ranitovic<ir...@gmail.com> wrote:
> After setting up a test table with about 250K rows, I wanted to enable
> lzo compression on it.
>
> In the shell, I disabled and altered the table and then ran major_compaction.
>
> This is the result:
>
> hbase(main):013: major_compact 'urls'
> 0 row(s) in 0.0460 seconds
>
> Does '0 rows' means that this did not work?
> (the size of the table on the disk did not change/reduce)
>
> Table info:
> hbase(main):019:0> describe 'urls'
> DESCRIPTION
> ENABLED
>
> false
> {NAME => 'urls', FAMILIES => [{NAME => 'data', VERSIONS => '1',
> COMPRE
> SSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
> =
> > 'false', BLOCKCACHE => 'true'}]}
> 1 row(s) in 0.0420 seconds
>
>
> Table Count:
> hbase(main):021:0> count 'urls'
> ....
> Current count: 251000, row: http://com,zz/
>
>
> From the master log:
> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,880 INFO
> com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl
> library
> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,891 INFO
> com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized
> native-lzo library
>
> I using 0.20. Thanks.
>
> i.
>