You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by kennyut <ke...@gmail.com> on 2015/01/27 23:54:40 UTC
After compression, the table folders under hdfs://hbase are empty
I tried to test HBase's data compression, I used two separate codes below:
non-compression code:
create 'SMT_KO1', {NAME=>'info', COMPRESSION=> 'NONE', VERSIONS => 5},
{NAME=>'usg', COMPRESSION=> 'NONE;, VERSIONS => 10},
SPLITS=> ['0','1', '2', '3', '4', '5', '6', '7', '8',
'9']
compression code:
create 'SMT_KO2', {NAME=>'info', COMPRESSION=>'lz4', VERSIONS => 5},
{NAME=>'usg', COMPRESSION=>'lz4', VERSIONS => 10},
SPLITS=> ['0','1', '2', '3', '4', '5', '6', '7', '8',
'9']
I can find two data folders under my hdfs://hbase path. One is SMT_KO1 and
the other is SMT_KO2.
The SMT_KO1 looks OK, the total folder size is around 3.1 Gb, but the
SMT_KO2 which is created from the compression code has only 1008 bytes.
Does someone know where do those files go? Both tables look ok if I open
them in Hue's HBase browser.
Any suggestion?
Thank you!
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/After-compression-the-table-folders-under-hdfs-hbase-are-empty-tp4067921.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: After compression, the table folders under hdfs://hbase are empty
Posted by Artem Ervits <ar...@gmail.com>.
I am sorry fat-fingers..
Artem Ervits
On Jan 29, 2015 8:40 PM, "Artem Ervits" <ar...@gmail.com> wrote:
> Artem Ervits
> On Jan 28, 2015 1:12 PM, "kennyut" <ke...@gmail.com> wrote:
>
>> In the /var/log/hbase folder, I can find two files:
>> 1. hbase-cmf-hbase-HBASERESTSERVER-ip-172-x-x-x.ec2.internal.log.out
>> 2. hbase-cmf-hbase-MASTER-ip-172-x-x-x.ec2.internal.log.out
>>
>> The '1' file was updated several days before, so it won't help.
>>
>> The '2' file only records the hbase log when I use the 'create' command in
>> 'hbase shell'. It shows nothing related to data loading. BTW, I use
>> happybase (python's hbase package) to load data into hbase.
>>
>> I cannot find any similar files in the hdfs://hbase folder, what I can
>> find
>> are the empty column family folders. But if I switch back to
>> compression='NONE' then all the files appear in those column family
>> folders.
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-hbase.679495.n3.nabble.com/After-compression-the-table-folders-under-hdfs-hbase-are-empty-tp4067921p4067947.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>
Re: After compression, the table folders under hdfs://hbase are empty
Posted by Artem Ervits <ar...@gmail.com>.
Artem Ervits
On Jan 28, 2015 1:12 PM, "kennyut" <ke...@gmail.com> wrote:
> In the /var/log/hbase folder, I can find two files:
> 1. hbase-cmf-hbase-HBASERESTSERVER-ip-172-x-x-x.ec2.internal.log.out
> 2. hbase-cmf-hbase-MASTER-ip-172-x-x-x.ec2.internal.log.out
>
> The '1' file was updated several days before, so it won't help.
>
> The '2' file only records the hbase log when I use the 'create' command in
> 'hbase shell'. It shows nothing related to data loading. BTW, I use
> happybase (python's hbase package) to load data into hbase.
>
> I cannot find any similar files in the hdfs://hbase folder, what I can find
> are the empty column family folders. But if I switch back to
> compression='NONE' then all the files appear in those column family
> folders.
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/After-compression-the-table-folders-under-hdfs-hbase-are-empty-tp4067921p4067947.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: After compression, the table folders under hdfs://hbase are
empty
Posted by kennyut <ke...@gmail.com>.
In the /var/log/hbase folder, I can find two files:
1. hbase-cmf-hbase-HBASERESTSERVER-ip-172-x-x-x.ec2.internal.log.out
2. hbase-cmf-hbase-MASTER-ip-172-x-x-x.ec2.internal.log.out
The '1' file was updated several days before, so it won't help.
The '2' file only records the hbase log when I use the 'create' command in
'hbase shell'. It shows nothing related to data loading. BTW, I use
happybase (python's hbase package) to load data into hbase.
I cannot find any similar files in the hdfs://hbase folder, what I can find
are the empty column family folders. But if I switch back to
compression='NONE' then all the files appear in those column family folders.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/After-compression-the-table-folders-under-hdfs-hbase-are-empty-tp4067921p4067947.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: After compression, the table folders under hdfs://hbase are
empty
Posted by kennyut <ke...@gmail.com>.
Thanks Jean and Ted. I think i've found it. It's under:
/var/log/hbase
I am looking into the file. Will post update after.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/After-compression-the-table-folders-under-hdfs-hbase-are-empty-tp4067921p4067946.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: After compression, the table folders under hdfs://hbase are empty
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Have you looked a the default /var/log directory to see what you have there?
JM
2015-01-28 9:11 GMT-08:00 Ted Yu <yu...@gmail.com>:
> I am not familiar with CDH deployment.
> Log onto the server where regionserver process is running and issue:
>
> ps aux | grep regionserver
>
> In the output look for '-Dhbase.log.dir=' which would tell you the location
> of the log.
>
> Cheers
>
> On Wed, Jan 28, 2015 at 7:37 AM, kennyut <ke...@gmail.com> wrote:
>
> > I can find the hbase version from hbase shell (please see below):
> >
> > hbase(main):001:0> version
> > 0.98.1-cdh5.1.3, rUnknown, Tue Sep 16 20:19:34 PDT 2014
> >
> > But where I can find my HBase region server log? I used all default
> option
> > when installing HBase, including all configurations.
> >
> > Thanks for your reply!
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/After-compression-the-table-folders-under-hdfs-hbase-are-empty-tp4067921p4067938.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
>
Re: After compression, the table folders under hdfs://hbase are empty
Posted by Ted Yu <yu...@gmail.com>.
I am not familiar with CDH deployment.
Log onto the server where regionserver process is running and issue:
ps aux | grep regionserver
In the output look for '-Dhbase.log.dir=' which would tell you the location
of the log.
Cheers
On Wed, Jan 28, 2015 at 7:37 AM, kennyut <ke...@gmail.com> wrote:
> I can find the hbase version from hbase shell (please see below):
>
> hbase(main):001:0> version
> 0.98.1-cdh5.1.3, rUnknown, Tue Sep 16 20:19:34 PDT 2014
>
> But where I can find my HBase region server log? I used all default option
> when installing HBase, including all configurations.
>
> Thanks for your reply!
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/After-compression-the-table-folders-under-hdfs-hbase-are-empty-tp4067921p4067938.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: After compression, the table folders under hdfs://hbase are
empty
Posted by kennyut <ke...@gmail.com>.
I can find the hbase version from hbase shell (please see below):
hbase(main):001:0> version
0.98.1-cdh5.1.3, rUnknown, Tue Sep 16 20:19:34 PDT 2014
But where I can find my HBase region server log? I used all default option
when installing HBase, including all configurations.
Thanks for your reply!
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/After-compression-the-table-folders-under-hdfs-hbase-are-empty-tp4067921p4067938.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: After compression, the table folders under hdfs://hbase are empty
Posted by Ted Yu <yu...@gmail.com>.
Which HBase release are you using ?
Have you checked region server log(s) and look for SMT_KO2 ?
Cheers
On Tue, Jan 27, 2015 at 2:54 PM, kennyut <ke...@gmail.com> wrote:
> I tried to test HBase's data compression, I used two separate codes below:
>
> non-compression code:
> create 'SMT_KO1', {NAME=>'info', COMPRESSION=> 'NONE', VERSIONS => 5},
> {NAME=>'usg', COMPRESSION=> 'NONE;, VERSIONS => 10},
> SPLITS=> ['0','1', '2', '3', '4', '5', '6', '7', '8',
> '9']
>
> compression code:
> create 'SMT_KO2', {NAME=>'info', COMPRESSION=>'lz4', VERSIONS => 5},
> {NAME=>'usg', COMPRESSION=>'lz4', VERSIONS => 10},
> SPLITS=> ['0','1', '2', '3', '4', '5', '6', '7', '8',
> '9']
>
> I can find two data folders under my hdfs://hbase path. One is SMT_KO1 and
> the other is SMT_KO2.
> The SMT_KO1 looks OK, the total folder size is around 3.1 Gb, but the
> SMT_KO2 which is created from the compression code has only 1008 bytes.
>
> Does someone know where do those files go? Both tables look ok if I open
> them in Hue's HBase browser.
> Any suggestion?
>
> Thank you!
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/After-compression-the-table-folders-under-hdfs-hbase-are-empty-tp4067921.html
> Sent from the HBase User mailing list archive at Nabble.com.
>