You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Upendra Yadav <up...@gmail.com> on 2014/02/25 08:07:37 UTC
Is hbase call delete request for old hfile(storeFile) file if
minor/major compaction happen for them
Is hbase call delete request for old hfile(storeFile) file if minor/major
compaction happen for them and generate new hfile and store it in hdfs.
for minor compaction how it will get size of every hfile. lets we have 20
no. of hfiles for a CF and minor compaction wants to perform. Will it
download all hfile and decide compaction have to do or not. Or it have
in-memory hfile's size information for all hfile.
Re: Is hbase call delete request for old hfile(storeFile) file if
minor/major compaction happen for them
Posted by Upendra Yadav <up...@gmail.com>.
will hbase call for delete old hfiles on hdfs? Since on minor/major
compaction that files already merged and after merging generate new hfile.
So old hfile will never be use.
On Tue, Feb 25, 2014 at 12:37 PM, Upendra Yadav <up...@gmail.com>wrote:
> Is hbase call delete request for old hfile(storeFile) file if minor/major
> compaction happen for them and generate new hfile and store it in hdfs.
>
> for minor compaction how it will get size of every hfile. lets we have 20
> no. of hfiles for a CF and minor compaction wants to perform. Will it
> download all hfile and decide compaction have to do or not. Or it have
> in-memory hfile's size information for all hfile.
>
>
>
Re: Is hbase call delete request for old hfile(storeFile) file if
minor/major compaction happen for them
Posted by Upendra Yadav <up...@gmail.com>.
Thanks a lot.... Jean & Bharath
It really helping me to get clear and deeper view on how hbase perform its
required operation...
Is that delete operation for old hfile happened by schedule?
On Tue, Feb 25, 2014 at 8:17 PM, Bharath Vissapragada <bharathv@cloudera.com
> wrote:
> Hi,
>
> I believe CatalogJanitor is for a different purpose (cleaning meta post
> splits). He should be looking at o.a.h.h.master.cleaner.* ! Anway files are
> deleted.
>
> - Bharath
>
>
> On Tue, Feb 25, 2014 at 5:43 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > Hi Upendra,
> >
> > Olds files are moved by HBase later. Take a look at
> > org.apache.hadoop.hbase.master.CatalogJanitor
> >
> > JM
> >
> >
> > 2014-02-25 6:25 GMT-05:00 Upendra Yadav <up...@gmail.com>:
> >
> > > Thanks...
> > >
> > >
> > > On Tue, Feb 25, 2014 at 2:18 PM, Bharath Vissapragada <
> > > bharathv@cloudera.com
> > > > wrote:
> > >
> > > > In short, it needn't "download" files, that metadata is stored and
> > > exposed
> > > > in the API of a StoreFile and regionserver just calls that method!
> > > >
> > > >
> > > > On Tue, Feb 25, 2014 at 2:16 PM, Bharath Vissapragada <
> > > > bharathv@cloudera.com
> > > > > wrote:
> > > >
> > > > > Not sure what you are asking but if you want to know the size of a
> > > store
> > > > > file, you can use this method,
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java#L1440
> > > > >
> > > > > One such usage is here,
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java#L292
> > > > >
> > > > > It gets the sizes of all the store files, I believe thats what you
> > are
> > > > > asking! Please correct me if I'm wrong,
> > > > >
> > > > >
> > > > > On Tue, Feb 25, 2014 at 12:37 PM, Upendra Yadav <
> > upendra1024@gmail.com
> > > > >wrote:
> > > > >
> > > > >> Is hbase call delete request for old hfile(storeFile) file if
> > > > minor/major
> > > > >> compaction happen for them and generate new hfile and store it in
> > > hdfs.
> > > > >>
> > > > >> for minor compaction how it will get size of every hfile. lets we
> > have
> > > > 20
> > > > >> no. of hfiles for a CF and minor compaction wants to perform. Will
> > it
> > > > >> download all hfile and decide compaction have to do or not. Or it
> > have
> > > > >> in-memory hfile's size information for all hfile.
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Bharath Vissapragada
> > > > > <http://www.cloudera.com>
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Bharath Vissapragada
> > > > <http://www.cloudera.com>
> > > >
> > >
> >
>
>
>
> --
> Bharath Vissapragada
> <http://www.cloudera.com>
>
Re: Is hbase call delete request for old hfile(storeFile) file if
minor/major compaction happen for them
Posted by Bharath Vissapragada <bh...@cloudera.com>.
Hi,
I believe CatalogJanitor is for a different purpose (cleaning meta post
splits). He should be looking at o.a.h.h.master.cleaner.* ! Anway files are
deleted.
- Bharath
On Tue, Feb 25, 2014 at 5:43 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Upendra,
>
> Olds files are moved by HBase later. Take a look at
> org.apache.hadoop.hbase.master.CatalogJanitor
>
> JM
>
>
> 2014-02-25 6:25 GMT-05:00 Upendra Yadav <up...@gmail.com>:
>
> > Thanks...
> >
> >
> > On Tue, Feb 25, 2014 at 2:18 PM, Bharath Vissapragada <
> > bharathv@cloudera.com
> > > wrote:
> >
> > > In short, it needn't "download" files, that metadata is stored and
> > exposed
> > > in the API of a StoreFile and regionserver just calls that method!
> > >
> > >
> > > On Tue, Feb 25, 2014 at 2:16 PM, Bharath Vissapragada <
> > > bharathv@cloudera.com
> > > > wrote:
> > >
> > > > Not sure what you are asking but if you want to know the size of a
> > store
> > > > file, you can use this method,
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java#L1440
> > > >
> > > > One such usage is here,
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java#L292
> > > >
> > > > It gets the sizes of all the store files, I believe thats what you
> are
> > > > asking! Please correct me if I'm wrong,
> > > >
> > > >
> > > > On Tue, Feb 25, 2014 at 12:37 PM, Upendra Yadav <
> upendra1024@gmail.com
> > > >wrote:
> > > >
> > > >> Is hbase call delete request for old hfile(storeFile) file if
> > > minor/major
> > > >> compaction happen for them and generate new hfile and store it in
> > hdfs.
> > > >>
> > > >> for minor compaction how it will get size of every hfile. lets we
> have
> > > 20
> > > >> no. of hfiles for a CF and minor compaction wants to perform. Will
> it
> > > >> download all hfile and decide compaction have to do or not. Or it
> have
> > > >> in-memory hfile's size information for all hfile.
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Bharath Vissapragada
> > > > <http://www.cloudera.com>
> > > >
> > >
> > >
> > >
> > > --
> > > Bharath Vissapragada
> > > <http://www.cloudera.com>
> > >
> >
>
--
Bharath Vissapragada
<http://www.cloudera.com>
Re: Is hbase call delete request for old hfile(storeFile) file if
minor/major compaction happen for them
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Upendra,
Olds files are moved by HBase later. Take a look at
org.apache.hadoop.hbase.master.CatalogJanitor
JM
2014-02-25 6:25 GMT-05:00 Upendra Yadav <up...@gmail.com>:
> Thanks...
>
>
> On Tue, Feb 25, 2014 at 2:18 PM, Bharath Vissapragada <
> bharathv@cloudera.com
> > wrote:
>
> > In short, it needn't "download" files, that metadata is stored and
> exposed
> > in the API of a StoreFile and regionserver just calls that method!
> >
> >
> > On Tue, Feb 25, 2014 at 2:16 PM, Bharath Vissapragada <
> > bharathv@cloudera.com
> > > wrote:
> >
> > > Not sure what you are asking but if you want to know the size of a
> store
> > > file, you can use this method,
> > >
> > >
> > >
> >
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java#L1440
> > >
> > > One such usage is here,
> > >
> > >
> > >
> >
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java#L292
> > >
> > > It gets the sizes of all the store files, I believe thats what you are
> > > asking! Please correct me if I'm wrong,
> > >
> > >
> > > On Tue, Feb 25, 2014 at 12:37 PM, Upendra Yadav <upendra1024@gmail.com
> > >wrote:
> > >
> > >> Is hbase call delete request for old hfile(storeFile) file if
> > minor/major
> > >> compaction happen for them and generate new hfile and store it in
> hdfs.
> > >>
> > >> for minor compaction how it will get size of every hfile. lets we have
> > 20
> > >> no. of hfiles for a CF and minor compaction wants to perform. Will it
> > >> download all hfile and decide compaction have to do or not. Or it have
> > >> in-memory hfile's size information for all hfile.
> > >>
> > >
> > >
> > >
> > > --
> > > Bharath Vissapragada
> > > <http://www.cloudera.com>
> > >
> >
> >
> >
> > --
> > Bharath Vissapragada
> > <http://www.cloudera.com>
> >
>
Re: Is hbase call delete request for old hfile(storeFile) file if
minor/major compaction happen for them
Posted by Upendra Yadav <up...@gmail.com>.
Thanks...
On Tue, Feb 25, 2014 at 2:18 PM, Bharath Vissapragada <bharathv@cloudera.com
> wrote:
> In short, it needn't "download" files, that metadata is stored and exposed
> in the API of a StoreFile and regionserver just calls that method!
>
>
> On Tue, Feb 25, 2014 at 2:16 PM, Bharath Vissapragada <
> bharathv@cloudera.com
> > wrote:
>
> > Not sure what you are asking but if you want to know the size of a store
> > file, you can use this method,
> >
> >
> >
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java#L1440
> >
> > One such usage is here,
> >
> >
> >
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java#L292
> >
> > It gets the sizes of all the store files, I believe thats what you are
> > asking! Please correct me if I'm wrong,
> >
> >
> > On Tue, Feb 25, 2014 at 12:37 PM, Upendra Yadav <upendra1024@gmail.com
> >wrote:
> >
> >> Is hbase call delete request for old hfile(storeFile) file if
> minor/major
> >> compaction happen for them and generate new hfile and store it in hdfs.
> >>
> >> for minor compaction how it will get size of every hfile. lets we have
> 20
> >> no. of hfiles for a CF and minor compaction wants to perform. Will it
> >> download all hfile and decide compaction have to do or not. Or it have
> >> in-memory hfile's size information for all hfile.
> >>
> >
> >
> >
> > --
> > Bharath Vissapragada
> > <http://www.cloudera.com>
> >
>
>
>
> --
> Bharath Vissapragada
> <http://www.cloudera.com>
>
Re: Is hbase call delete request for old hfile(storeFile) file if
minor/major compaction happen for them
Posted by Bharath Vissapragada <bh...@cloudera.com>.
In short, it needn't "download" files, that metadata is stored and exposed
in the API of a StoreFile and regionserver just calls that method!
On Tue, Feb 25, 2014 at 2:16 PM, Bharath Vissapragada <bharathv@cloudera.com
> wrote:
> Not sure what you are asking but if you want to know the size of a store
> file, you can use this method,
>
>
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java#L1440
>
> One such usage is here,
>
>
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java#L292
>
> It gets the sizes of all the store files, I believe thats what you are
> asking! Please correct me if I'm wrong,
>
>
> On Tue, Feb 25, 2014 at 12:37 PM, Upendra Yadav <up...@gmail.com>wrote:
>
>> Is hbase call delete request for old hfile(storeFile) file if minor/major
>> compaction happen for them and generate new hfile and store it in hdfs.
>>
>> for minor compaction how it will get size of every hfile. lets we have 20
>> no. of hfiles for a CF and minor compaction wants to perform. Will it
>> download all hfile and decide compaction have to do or not. Or it have
>> in-memory hfile's size information for all hfile.
>>
>
>
>
> --
> Bharath Vissapragada
> <http://www.cloudera.com>
>
--
Bharath Vissapragada
<http://www.cloudera.com>
Re: Is hbase call delete request for old hfile(storeFile) file if
minor/major compaction happen for them
Posted by Bharath Vissapragada <bh...@cloudera.com>.
Not sure what you are asking but if you want to know the size of a store
file, you can use this method,
https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java#L1440
One such usage is here,
https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java#L292
It gets the sizes of all the store files, I believe thats what you are
asking! Please correct me if I'm wrong,
On Tue, Feb 25, 2014 at 12:37 PM, Upendra Yadav <up...@gmail.com>wrote:
> Is hbase call delete request for old hfile(storeFile) file if minor/major
> compaction happen for them and generate new hfile and store it in hdfs.
>
> for minor compaction how it will get size of every hfile. lets we have 20
> no. of hfiles for a CF and minor compaction wants to perform. Will it
> download all hfile and decide compaction have to do or not. Or it have
> in-memory hfile's size information for all hfile.
>
--
Bharath Vissapragada
<http://www.cloudera.com>