You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "sasha.maksimenko" <sa...@gmail.com> on 2011/08/25 09:17:08 UTC
delete does not free hdd space
hi!
I need to free hdd space, so I want to delete some unimportant rows
I'm using Thrift call
deleteAllRow(TABLE, ROWID);
to perform deletion of about 10 000 row (when I loaded them it was about
30Gb). When I perform this operation I checked thru hbase shell that
rows are deleted, but used space of the disk is not changed
I performed compact and major_compact operations (they returned 0
processed rows) but space is not changed too.
So quesition what should I also do to indicate hbase physically remove
this rows
Re: delete does not free hdd space
Posted by Rohan Rai <ro...@inmobi.com>.
In addendum
region which are empty (caused due to deletion activity)
How to purge them
Regards
Rohan
On Thursday 25 August 2011 12:47 PM, sasha.maksimenko wrote:
> hi!
> I need to free hdd space, so I want to delete some unimportant rows
> I'm using Thrift call
> deleteAllRow(TABLE, ROWID);
> to perform deletion of about 10 000 row (when I loaded them it was
> about 30Gb). When I perform this operation I checked thru hbase shell
> that rows are deleted, but used space of the disk is not changed
> I performed compact and major_compact operations (they returned 0
> processed rows) but space is not changed too.
>
> So quesition what should I also do to indicate hbase physically remove
> this rows
>
________________________________
The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
Re: delete does not free hdd space
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Please don't hijack threads, this isn't relevant to the current
discussion. Please start a new one and, I might add, clarify your
question a bit :)
On Fri, Aug 26, 2011 at 9:44 AM, Rohan Rai <ro...@inmobi.com> wrote:
> n how to merge regions, or un register regions
>
> Regards
> Rohan
>
> On Friday 26 August 2011 09:42 PM, Buttler, David wrote:
>>
>> Major_compact is an asynchronous command. It may take some time for space
>> to be freed. If you are watching your data node logs you should be able to
>> see blocks being deleted. If you are on the local file system, you should
>> be table to periodically do a 'du -sh' command on the hbase table directory
>> and see the used space diminish.
>>
>> Dave
>>
>> -----Original Message-----
>> From: Joel Halbert [mailto:joel@su3analytics.com]
>> Sent: Friday, August 26, 2011 2:37 AM
>> To: user@hbase.apache.org
>> Subject: Re: delete does not free hdd space
>>
>> Hi,
>>
>> I experience exactly the same thing.
>>
>> After deleting many rows I do not see a decrease in size of data on
>> disk.
>>
>> When I run a major compact manually from the shell it outputs the same
>> as for Sasha:
>>
>> 0 row(s) in 0.9210 seconds
>>
>> I am running HBase standalone (without hadoop) on local disk.
>>
>> On Fri, 2011-08-26 at 10:42 +0300, sasha.maksimenko wrote:
>>>
>>> hi!
>>> hm
>>> I've deleted and after that run major_compact
>>>
>>> HBase Shell; enter 'help<RETURN>' for list of supported commands.
>>> Version: 0.20.6, r965666, Mon Jul 19 15:48:07 PDT 2010
>>> hbase(main):001:0> major_compact 'task'
>>> 0 row(s) in 0.9210 seconds
>>> hbase(main):002:0>
>>>
>>>
>>> is it could be a problem that I use local file system - not HDFS?
>>>
>>>
>>>> If you run a major_compact from the shell that should purge out deleted
>>>> records. By default the major compaction happens on every table every 24
>>>> hours.
>>>>
>>>> -chris
>>>>
>>>> On Aug 25, 2011, at 12:17 AM, sasha.maksimenko wrote:
>>>>
>>>>> hi!
>>>>> I need to free hdd space, so I want to delete some unimportant rows
>>>>> I'm using Thrift call
>>>>> deleteAllRow(TABLE, ROWID);
>>>>> to perform deletion of about 10 000 row (when I loaded them it was
>>>>> about 30Gb). When I perform this operation I checked thru hbase shell that
>>>>> rows are deleted, but used space of the disk is not changed
>>>>> I performed compact and major_compact operations (they returned 0
>>>>> processed rows) but space is not changed too.
>>>>>
>>>>> So quesition what should I also do to indicate hbase physically remove
>>>>> this rows
>>
>
>
> ________________________________
>
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify us
> immediately by responding to this email and then delete it from your system.
> The firm is neither liable for the proper and complete transmission of the
> information contained in this communication nor for any delay in its
> receipt.
>
Re: delete does not free hdd space
Posted by Rohan Rai <ro...@inmobi.com>.
n how to merge regions, or un register regions
Regards
Rohan
On Friday 26 August 2011 09:42 PM, Buttler, David wrote:
> Major_compact is an asynchronous command. It may take some time for space to be freed. If you are watching your data node logs you should be able to see blocks being deleted. If you are on the local file system, you should be table to periodically do a 'du -sh' command on the hbase table directory and see the used space diminish.
>
> Dave
>
> -----Original Message-----
> From: Joel Halbert [mailto:joel@su3analytics.com]
> Sent: Friday, August 26, 2011 2:37 AM
> To: user@hbase.apache.org
> Subject: Re: delete does not free hdd space
>
> Hi,
>
> I experience exactly the same thing.
>
> After deleting many rows I do not see a decrease in size of data on
> disk.
>
> When I run a major compact manually from the shell it outputs the same
> as for Sasha:
>
> 0 row(s) in 0.9210 seconds
>
> I am running HBase standalone (without hadoop) on local disk.
>
> On Fri, 2011-08-26 at 10:42 +0300, sasha.maksimenko wrote:
>> hi!
>> hm
>> I've deleted and after that run major_compact
>>
>> HBase Shell; enter 'help<RETURN>' for list of supported commands.
>> Version: 0.20.6, r965666, Mon Jul 19 15:48:07 PDT 2010
>> hbase(main):001:0> major_compact 'task'
>> 0 row(s) in 0.9210 seconds
>> hbase(main):002:0>
>>
>>
>> is it could be a problem that I use local file system - not HDFS?
>>
>>
>>> If you run a major_compact from the shell that should purge out deleted records. By default the major compaction happens on every table every 24 hours.
>>>
>>> -chris
>>>
>>> On Aug 25, 2011, at 12:17 AM, sasha.maksimenko wrote:
>>>
>>>> hi!
>>>> I need to free hdd space, so I want to delete some unimportant rows
>>>> I'm using Thrift call
>>>> deleteAllRow(TABLE, ROWID);
>>>> to perform deletion of about 10 000 row (when I loaded them it was about 30Gb). When I perform this operation I checked thru hbase shell that rows are deleted, but used space of the disk is not changed
>>>> I performed compact and major_compact operations (they returned 0 processed rows) but space is not changed too.
>>>>
>>>> So quesition what should I also do to indicate hbase physically remove this rows
>
________________________________
The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
Re: delete does not free hdd space
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Everything Dave said is right, but I might also add that since edits
are done in the Memstores and that forcing a major compaction
_doesn't_ force flush the Memstores, the deletes won't be taken into
account.
HBase also keeps all other sorts of data around, for example look at
the .logs directory. That's where the write-ahead-logs are kept and
they aren't cleaned that often (it's a matter of time, if you don't
insert anything then one log will get cleaned per hour).
J-D
On Fri, Aug 26, 2011 at 9:12 AM, Buttler, David <bu...@llnl.gov> wrote:
> Major_compact is an asynchronous command. It may take some time for space to be freed. If you are watching your data node logs you should be able to see blocks being deleted. If you are on the local file system, you should be table to periodically do a 'du -sh' command on the hbase table directory and see the used space diminish.
>
> Dave
>
> -----Original Message-----
> From: Joel Halbert [mailto:joel@su3analytics.com]
> Sent: Friday, August 26, 2011 2:37 AM
> To: user@hbase.apache.org
> Subject: Re: delete does not free hdd space
>
> Hi,
>
> I experience exactly the same thing.
>
> After deleting many rows I do not see a decrease in size of data on
> disk.
>
> When I run a major compact manually from the shell it outputs the same
> as for Sasha:
>
> 0 row(s) in 0.9210 seconds
>
> I am running HBase standalone (without hadoop) on local disk.
>
> On Fri, 2011-08-26 at 10:42 +0300, sasha.maksimenko wrote:
>> hi!
>> hm
>> I've deleted and after that run major_compact
>>
>> HBase Shell; enter 'help<RETURN>' for list of supported commands.
>> Version: 0.20.6, r965666, Mon Jul 19 15:48:07 PDT 2010
>> hbase(main):001:0> major_compact 'task'
>> 0 row(s) in 0.9210 seconds
>> hbase(main):002:0>
>>
>>
>> is it could be a problem that I use local file system - not HDFS?
>>
>>
>> > If you run a major_compact from the shell that should purge out deleted records. By default the major compaction happens on every table every 24 hours.
>> >
>> > -chris
>> >
>> > On Aug 25, 2011, at 12:17 AM, sasha.maksimenko wrote:
>> >
>> >> hi!
>> >> I need to free hdd space, so I want to delete some unimportant rows
>> >> I'm using Thrift call
>> >> deleteAllRow(TABLE, ROWID);
>> >> to perform deletion of about 10 000 row (when I loaded them it was about 30Gb). When I perform this operation I checked thru hbase shell that rows are deleted, but used space of the disk is not changed
>> >> I performed compact and major_compact operations (they returned 0 processed rows) but space is not changed too.
>> >>
>> >> So quesition what should I also do to indicate hbase physically remove this rows
>>
>
>
>
RE: delete does not free hdd space
Posted by "Buttler, David" <bu...@llnl.gov>.
Major_compact is an asynchronous command. It may take some time for space to be freed. If you are watching your data node logs you should be able to see blocks being deleted. If you are on the local file system, you should be table to periodically do a 'du -sh' command on the hbase table directory and see the used space diminish.
Dave
-----Original Message-----
From: Joel Halbert [mailto:joel@su3analytics.com]
Sent: Friday, August 26, 2011 2:37 AM
To: user@hbase.apache.org
Subject: Re: delete does not free hdd space
Hi,
I experience exactly the same thing.
After deleting many rows I do not see a decrease in size of data on
disk.
When I run a major compact manually from the shell it outputs the same
as for Sasha:
0 row(s) in 0.9210 seconds
I am running HBase standalone (without hadoop) on local disk.
On Fri, 2011-08-26 at 10:42 +0300, sasha.maksimenko wrote:
> hi!
> hm
> I've deleted and after that run major_compact
>
> HBase Shell; enter 'help<RETURN>' for list of supported commands.
> Version: 0.20.6, r965666, Mon Jul 19 15:48:07 PDT 2010
> hbase(main):001:0> major_compact 'task'
> 0 row(s) in 0.9210 seconds
> hbase(main):002:0>
>
>
> is it could be a problem that I use local file system - not HDFS?
>
>
> > If you run a major_compact from the shell that should purge out deleted records. By default the major compaction happens on every table every 24 hours.
> >
> > -chris
> >
> > On Aug 25, 2011, at 12:17 AM, sasha.maksimenko wrote:
> >
> >> hi!
> >> I need to free hdd space, so I want to delete some unimportant rows
> >> I'm using Thrift call
> >> deleteAllRow(TABLE, ROWID);
> >> to perform deletion of about 10 000 row (when I loaded them it was about 30Gb). When I perform this operation I checked thru hbase shell that rows are deleted, but used space of the disk is not changed
> >> I performed compact and major_compact operations (they returned 0 processed rows) but space is not changed too.
> >>
> >> So quesition what should I also do to indicate hbase physically remove this rows
>
Re: delete does not free hdd space
Posted by Joel Halbert <jo...@su3analytics.com>.
Hi,
I experience exactly the same thing.
After deleting many rows I do not see a decrease in size of data on
disk.
When I run a major compact manually from the shell it outputs the same
as for Sasha:
0 row(s) in 0.9210 seconds
I am running HBase standalone (without hadoop) on local disk.
On Fri, 2011-08-26 at 10:42 +0300, sasha.maksimenko wrote:
> hi!
> hm
> I've deleted and after that run major_compact
>
> HBase Shell; enter 'help<RETURN>' for list of supported commands.
> Version: 0.20.6, r965666, Mon Jul 19 15:48:07 PDT 2010
> hbase(main):001:0> major_compact 'task'
> 0 row(s) in 0.9210 seconds
> hbase(main):002:0>
>
>
> is it could be a problem that I use local file system - not HDFS?
>
>
> > If you run a major_compact from the shell that should purge out deleted records. By default the major compaction happens on every table every 24 hours.
> >
> > -chris
> >
> > On Aug 25, 2011, at 12:17 AM, sasha.maksimenko wrote:
> >
> >> hi!
> >> I need to free hdd space, so I want to delete some unimportant rows
> >> I'm using Thrift call
> >> deleteAllRow(TABLE, ROWID);
> >> to perform deletion of about 10 000 row (when I loaded them it was about 30Gb). When I perform this operation I checked thru hbase shell that rows are deleted, but used space of the disk is not changed
> >> I performed compact and major_compact operations (they returned 0 processed rows) but space is not changed too.
> >>
> >> So quesition what should I also do to indicate hbase physically remove this rows
>
Re: delete does not free hdd space
Posted by "sasha.maksimenko" <sa...@gmail.com>.
hi!
hm
I've deleted and after that run major_compact
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Version: 0.20.6, r965666, Mon Jul 19 15:48:07 PDT 2010
hbase(main):001:0> major_compact 'task'
0 row(s) in 0.9210 seconds
hbase(main):002:0>
is it could be a problem that I use local file system - not HDFS?
> If you run a major_compact from the shell that should purge out deleted records. By default the major compaction happens on every table every 24 hours.
>
> -chris
>
> On Aug 25, 2011, at 12:17 AM, sasha.maksimenko wrote:
>
>> hi!
>> I need to free hdd space, so I want to delete some unimportant rows
>> I'm using Thrift call
>> deleteAllRow(TABLE, ROWID);
>> to perform deletion of about 10 000 row (when I loaded them it was about 30Gb). When I perform this operation I checked thru hbase shell that rows are deleted, but used space of the disk is not changed
>> I performed compact and major_compact operations (they returned 0 processed rows) but space is not changed too.
>>
>> So quesition what should I also do to indicate hbase physically remove this rows
Re: delete does not free hdd space
Posted by Chris Tarnas <cf...@email.com>.
If you run a major_compact from the shell that should purge out deleted records. By default the major compaction happens on every table every 24 hours.
-chris
On Aug 25, 2011, at 12:17 AM, sasha.maksimenko wrote:
> hi!
> I need to free hdd space, so I want to delete some unimportant rows
> I'm using Thrift call
> deleteAllRow(TABLE, ROWID);
> to perform deletion of about 10 000 row (when I loaded them it was about 30Gb). When I perform this operation I checked thru hbase shell that rows are deleted, but used space of the disk is not changed
> I performed compact and major_compact operations (they returned 0 processed rows) but space is not changed too.
>
> So quesition what should I also do to indicate hbase physically remove this rows