You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vikram Singh Chandel <vi...@gmail.com> on 2014/02/10 13:31:14 UTC
How to get HBase table size using API
Hi
The requirement is to get the hbase table size (using API) and have to save
this size for each table in a metadata table .
i tried hdfs command to check table size but need api method (if available)
Hadoop fs -du -h hdfs://
Thanks
--
*Regards*
*VIKRAM SINGH CHANDEL*
Please do not print this email unless it is absolutely necessary,Reduce.
Reuse. Recycle. Save our planet.
Re: How to get HBase table size using API
Posted by Vikram Singh Chandel <vi...@gmail.com>.
Hi Lucas
Thanks a lot for your help i got what i required, really appreciate you for
giving your time on the issue.
On Tue, Feb 18, 2014 at 3:56 PM, Lukas Nalezenec <
lukas.nalezenec@firma.seznam.cz> wrote:
> Hi,
> Add this import:
> import org.apache.hadoop.hbase.HServerLoad;
>
> And change names of classes:
> ServerLoad -> HServerLoad
> RegionLoad -> HServerLoad.RegionLoad
>
> Lukas
>
>
> On 18.2.2014 11:19, Vikram Singh Chandel wrote:
>
>> Hi Lucas
>> As you said that RegionSizeCalculator is developed on top of 0.94, the
>> class has interdependencies vig
>> import org.apache.hadoop.hbase.RegionLoad;
>> import org.apache.hadoop.hbase.ServerLoad;
>>
>> unable to find these classes in 0.94.X
>>
>> are these classes available in 0.94 under some other package,
>>
>>
>>
>> On Tue, Feb 11, 2014 at 3:12 PM, Lukas Nalezenec <
>> lukas.nalezenec@firma.seznam.cz> wrote:
>>
>> Hi,
>>>
>>> I am hbase newbie, maybe there is simpler solution but this will work. I
>>> tried estimating size using HDFS but it is not best solution(see link
>>> [1]);
>>>
>>> You dont need to work with TableSplits., look at class
>>> org.apache.hadoop.hbase.util.RegionSizeCalculator.
>>> It can do what you need. Create instance of this class, than call method
>>> getRegionSizeMap() and sum all values in map. Note that the size contains
>>> only storeFile sizes, not memStore sizes.
>>> If you need customize behaviour of this class, just copy the code and
>>> change it.
>>>
>>> This class will be in version 0.98 but it was developed on 0.94 - it will
>>> work but you will have to change some java imports.
>>>
>>>
>>> [1]
>>> https://issues.apache.org/jira/browse/HBASE-10413?
>>> focusedCommentId=13889745&page=com.atlassian.jira.
>>> plugin.system.issuetabpanels:comment-tabpanel#comment-13889745
>>>
>>> Lukas
>>>
>>>
>>>
>>> On 11.2.2014 08:14, Vikram Singh Chandel wrote:
>>>
>>> Hi Lukas
>>>>
>>>> the table split constructor expects startRow, endRow and location we
>>>> won't
>>>> be having info about any of these.
>>>> Moreover we require table size as a whole, not split size.
>>>>
>>>> We will use the table size to look for a threshold breach in metadata
>>>> table, if breached we have to trigger a delete operation on to the
>>>> table(of
>>>> which threshold is breached) to delete LRU records until table size is
>>>> within limit (~ 50-60Gb)
>>>>
>>>>
>>>> On Mon, Feb 10, 2014 at 6:01 PM, Vikram Singh Chandel <
>>>> vikramsinghchandel@gmail.com> wrote:
>>>>
>>>> Hi
>>>>
>>>>> The requirement is to get the hbase table size (using API) and have to
>>>>> save this size for each table in a metadata table .
>>>>>
>>>>> i tried hdfs command to check table size but need api method (if
>>>>> available)
>>>>>
>>>>> Hadoop fs -du -h hdfs://
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> --
>>>>> *Regards*
>>>>>
>>>>> *VIKRAM SINGH CHANDEL*
>>>>>
>>>>>
>>>>> Please do not print this email unless it is absolutely
>>>>> necessary,Reduce.
>>>>> Reuse. Recycle. Save our planet.
>>>>>
>>>>>
>>>>>
>>>>
>>
>
--
*Regards*
*VIKRAM SINGH CHANDEL*
Please do not print this email unless it is absolutely necessary,Reduce.
Reuse. Recycle. Save our planet.
Re: How to get HBase table size using API
Posted by Lukas Nalezenec <lu...@firma.seznam.cz>.
Hi,
Add this import:
import org.apache.hadoop.hbase.HServerLoad;
And change names of classes:
ServerLoad -> HServerLoad
RegionLoad -> HServerLoad.RegionLoad
Lukas
On 18.2.2014 11:19, Vikram Singh Chandel wrote:
> Hi Lucas
> As you said that RegionSizeCalculator is developed on top of 0.94, the
> class has interdependencies vig
> import org.apache.hadoop.hbase.RegionLoad;
> import org.apache.hadoop.hbase.ServerLoad;
>
> unable to find these classes in 0.94.X
>
> are these classes available in 0.94 under some other package,
>
>
>
> On Tue, Feb 11, 2014 at 3:12 PM, Lukas Nalezenec <
> lukas.nalezenec@firma.seznam.cz> wrote:
>
>> Hi,
>>
>> I am hbase newbie, maybe there is simpler solution but this will work. I
>> tried estimating size using HDFS but it is not best solution(see link [1]);
>>
>> You dont need to work with TableSplits., look at class
>> org.apache.hadoop.hbase.util.RegionSizeCalculator.
>> It can do what you need. Create instance of this class, than call method
>> getRegionSizeMap() and sum all values in map. Note that the size contains
>> only storeFile sizes, not memStore sizes.
>> If you need customize behaviour of this class, just copy the code and
>> change it.
>>
>> This class will be in version 0.98 but it was developed on 0.94 - it will
>> work but you will have to change some java imports.
>>
>>
>> [1]
>> https://issues.apache.org/jira/browse/HBASE-10413?
>> focusedCommentId=13889745&page=com.atlassian.jira.
>> plugin.system.issuetabpanels:comment-tabpanel#comment-13889745
>>
>> Lukas
>>
>>
>>
>> On 11.2.2014 08:14, Vikram Singh Chandel wrote:
>>
>>> Hi Lukas
>>>
>>> the table split constructor expects startRow, endRow and location we won't
>>> be having info about any of these.
>>> Moreover we require table size as a whole, not split size.
>>>
>>> We will use the table size to look for a threshold breach in metadata
>>> table, if breached we have to trigger a delete operation on to the
>>> table(of
>>> which threshold is breached) to delete LRU records until table size is
>>> within limit (~ 50-60Gb)
>>>
>>>
>>> On Mon, Feb 10, 2014 at 6:01 PM, Vikram Singh Chandel <
>>> vikramsinghchandel@gmail.com> wrote:
>>>
>>> Hi
>>>> The requirement is to get the hbase table size (using API) and have to
>>>> save this size for each table in a metadata table .
>>>>
>>>> i tried hdfs command to check table size but need api method (if
>>>> available)
>>>>
>>>> Hadoop fs -du -h hdfs://
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> --
>>>> *Regards*
>>>>
>>>> *VIKRAM SINGH CHANDEL*
>>>>
>>>>
>>>> Please do not print this email unless it is absolutely necessary,Reduce.
>>>> Reuse. Recycle. Save our planet.
>>>>
>>>>
>>>
>
Re: How to get HBase table size using API
Posted by Vikram Singh Chandel <vi...@gmail.com>.
Hi Lucas
As you said that RegionSizeCalculator is developed on top of 0.94, the
class has interdependencies vig
import org.apache.hadoop.hbase.RegionLoad;
import org.apache.hadoop.hbase.ServerLoad;
unable to find these classes in 0.94.X
are these classes available in 0.94 under some other package,
On Tue, Feb 11, 2014 at 3:12 PM, Lukas Nalezenec <
lukas.nalezenec@firma.seznam.cz> wrote:
>
> Hi,
>
> I am hbase newbie, maybe there is simpler solution but this will work. I
> tried estimating size using HDFS but it is not best solution(see link [1]);
>
> You dont need to work with TableSplits., look at class
> org.apache.hadoop.hbase.util.RegionSizeCalculator.
> It can do what you need. Create instance of this class, than call method
> getRegionSizeMap() and sum all values in map. Note that the size contains
> only storeFile sizes, not memStore sizes.
> If you need customize behaviour of this class, just copy the code and
> change it.
>
> This class will be in version 0.98 but it was developed on 0.94 - it will
> work but you will have to change some java imports.
>
>
> [1]
> https://issues.apache.org/jira/browse/HBASE-10413?
> focusedCommentId=13889745&page=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-13889745
>
> Lukas
>
>
>
> On 11.2.2014 08:14, Vikram Singh Chandel wrote:
>
>> Hi Lukas
>>
>> the table split constructor expects startRow, endRow and location we won't
>> be having info about any of these.
>> Moreover we require table size as a whole, not split size.
>>
>> We will use the table size to look for a threshold breach in metadata
>> table, if breached we have to trigger a delete operation on to the
>> table(of
>> which threshold is breached) to delete LRU records until table size is
>> within limit (~ 50-60Gb)
>>
>>
>> On Mon, Feb 10, 2014 at 6:01 PM, Vikram Singh Chandel <
>> vikramsinghchandel@gmail.com> wrote:
>>
>> Hi
>>>
>>> The requirement is to get the hbase table size (using API) and have to
>>> save this size for each table in a metadata table .
>>>
>>> i tried hdfs command to check table size but need api method (if
>>> available)
>>>
>>> Hadoop fs -du -h hdfs://
>>>
>>>
>>>
>>> Thanks
>>>
>>> --
>>> *Regards*
>>>
>>> *VIKRAM SINGH CHANDEL*
>>>
>>>
>>> Please do not print this email unless it is absolutely necessary,Reduce.
>>> Reuse. Recycle. Save our planet.
>>>
>>>
>>
>>
>
--
*Regards*
*VIKRAM SINGH CHANDEL*
Please do not print this email unless it is absolutely necessary,Reduce.
Reuse. Recycle. Save our planet.
Re: How to get HBase table size using API
Posted by Lukas Nalezenec <lu...@firma.seznam.cz>.
Hi,
I am hbase newbie, maybe there is simpler solution but this will work. I
tried estimating size using HDFS but it is not best solution(see link [1]);
You dont need to work with TableSplits., look at class
org.apache.hadoop.hbase.util.RegionSizeCalculator.
It can do what you need. Create instance of this class, than call method
getRegionSizeMap() and sum all values in map. Note that the size
contains only storeFile sizes, not memStore sizes.
If you need customize behaviour of this class, just copy the code and
change it.
This class will be in version 0.98 but it was developed on 0.94 - it
will work but you will have to change some java imports.
[1]
https://issues.apache.org/jira/browse/HBASE-10413?focusedCommentId=13889745&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13889745
Lukas
On 11.2.2014 08:14, Vikram Singh Chandel wrote:
> Hi Lukas
>
> the table split constructor expects startRow, endRow and location we won't
> be having info about any of these.
> Moreover we require table size as a whole, not split size.
>
> We will use the table size to look for a threshold breach in metadata
> table, if breached we have to trigger a delete operation on to the table(of
> which threshold is breached) to delete LRU records until table size is
> within limit (~ 50-60Gb)
>
>
> On Mon, Feb 10, 2014 at 6:01 PM, Vikram Singh Chandel <
> vikramsinghchandel@gmail.com> wrote:
>
>> Hi
>>
>> The requirement is to get the hbase table size (using API) and have to
>> save this size for each table in a metadata table .
>>
>> i tried hdfs command to check table size but need api method (if available)
>>
>> Hadoop fs -du -h hdfs://
>>
>>
>>
>> Thanks
>>
>> --
>> *Regards*
>>
>> *VIKRAM SINGH CHANDEL*
>>
>> Please do not print this email unless it is absolutely necessary,Reduce.
>> Reuse. Recycle. Save our planet.
>>
>
>
Re: How to get HBase table size using API
Posted by Vikram Singh Chandel <vi...@gmail.com>.
Hi Lukas
the table split constructor expects startRow, endRow and location we won't
be having info about any of these.
Moreover we require table size as a whole, not split size.
We will use the table size to look for a threshold breach in metadata
table, if breached we have to trigger a delete operation on to the table(of
which threshold is breached) to delete LRU records until table size is
within limit (~ 50-60Gb)
On Mon, Feb 10, 2014 at 6:01 PM, Vikram Singh Chandel <
vikramsinghchandel@gmail.com> wrote:
> Hi
>
> The requirement is to get the hbase table size (using API) and have to
> save this size for each table in a metadata table .
>
> i tried hdfs command to check table size but need api method (if available)
>
> Hadoop fs -du -h hdfs://
>
>
>
> Thanks
>
> --
> *Regards*
>
> *VIKRAM SINGH CHANDEL*
>
> Please do not print this email unless it is absolutely necessary,Reduce.
> Reuse. Recycle. Save our planet.
>
--
*Regards*
*VIKRAM SINGH CHANDEL*
Please do not print this email unless it is absolutely necessary,Reduce.
Reuse. Recycle. Save our planet.
Re: How to get HBase table size using API
Posted by Ted Yu <yu...@gmail.com>.
HBASE-10413 has been resolved.
FYI
On Mon, Feb 10, 2014 at 4:34 AM, Lukas Nalezenec <
lukas.nalezenec@firma.seznam.cz> wrote:
> Hi,
> I am just working on something similar.
> See ticket https://issues.apache.org/jira/browse/HBASE-10413 . I am going
> to close the ticket today.
> Lukas
>
> On 10.2.2014 13:31, Vikram Singh Chandel wrote:
>
>> Hi
>>
>> The requirement is to get the hbase table size (using API) and have to
>> save
>> this size for each table in a metadata table .
>>
>> i tried hdfs command to check table size but need api method (if
>> available)
>>
>> Hadoop fs -du -h hdfs://
>>
>>
>>
>> Thanks
>>
>>
>
Re: How to get HBase table size using API
Posted by Lukas Nalezenec <lu...@firma.seznam.cz>.
Hi,
I am just working on something similar.
See ticket https://issues.apache.org/jira/browse/HBASE-10413 . I am
going to close the ticket today.
Lukas
On 10.2.2014 13:31, Vikram Singh Chandel wrote:
> Hi
>
> The requirement is to get the hbase table size (using API) and have to save
> this size for each table in a metadata table .
>
> i tried hdfs command to check table size but need api method (if available)
>
> Hadoop fs -du -h hdfs://
>
>
>
> Thanks
>