RE: FW: Reg: The HLog that is created in region creation

[Ramkrishna S Vasudevan <ra...@huawei.com>]

Hi all,

Sorry Ted for not sending to the dev@ list.

Few more queries related to splitting

1. As per the flow when compaction is happening there may be few more store
files created due to flushing.
2. Suppose initially 3 Store files have been selected for compaction at the
end of compaction process i will get 1 storefile.
Parallely due to flushing some 3 store files were created.

So after step 2 my total store files in 4.

3. Now inorder to get the midkey for splitting we iterate through all the
store files and find the midkey from the largest store file(Correct me if am
wrong).
4. this largest store file may be the store file that was created as part of
compaction.  

Now how does the midkey selected will encompass all the 4 store files keys?

Why i have this doubt is while splitting all the store files are moved into
the new regions created as part of split.

So all the 4 store files are now moved to both the new regions created.

Pls help me in understanding this flow.

Thanks in advance.

Regards
Ram




****************************************************************************
***********
This e-mail and attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above. Any use of the information contained herein in any way (including,
but not limited to, total or partial disclosure, reproduction, or
dissemination) by persons other than the intended recipient's) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!


-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com] 
Sent: Tuesday, July 05, 2011 11:01 AM
To: ramakrishnas@huawei.com
Cc: user@hbase.apache.org
Subject: Re: FW: Reg: The HLog that is created in region creation

Please include dev@ so that other people would be able to answer your
question.
>> and Daugter A2(Midkey to startkey).
Should read (Midkey to endkey).

For #1 below, you're right that splitting benefits retrieval more than
writes.
For #2, I guess you may have read CompactSplitThread code where compacting
stores in a region happens before splitting.

Cheers

On Mon, Jul 4, 2011 at 9:26 PM, Ramkrishna S Vasudevan <
ramakrishnas@huawei.com> wrote:

> Dear Ted
>
> Thanks for your reply.
>
> I have few questions on Splitting of regions.
>
> I will tell my observations, may be they are not fully correct. Pls
correct
> me if am wrong,
>
> Every time compaction results in a file greater than max file size region
> splits happen from the mid key.
>
> So Region A is now split into Daughter A1(Start to midkey) and Daugter
> A2(Midkey to startkey).
>
> 1. Why do we do a split ?
> I suppose it is because now a Region will now hold only a portion of the
> full data and hence retrieval will be easy.
>
> 2. Is there a possibility that during first compaction a region split
> happens, which results in 2 region creations.
>
> Now when new write request come to the parent region will it be
accomodated
> in the new daughter regions that were created.
> If it is the case, now how will the splitkeys be handled ?
>
> Pls do corect me if am wrong anywhere?
>
> Regards
> Ram
>
>
>
>
>
****************************************************************************
> ***********
> This e-mail and attachments contain confidential information from HUAWEI,
> which is intended only for the person or entity whose address is listed
> above. Any use of the information contained herein in any way (including,
> but not limited to, total or partial disclosure, reproduction, or
> dissemination) by persons other than the intended recipient's) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by
> phone or email immediately and delete it!
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Tuesday, July 05, 2011 9:38 AM
> To: user@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: Reg: The HLog that is created in region creation
>
> Not really used.
> See hbase-4010
>
>
>
> On Jul 4, 2011, at 8:55 PM, Ramkrishna S Vasudevan <
> ramakrishnas@huawei.com>
> wrote:
>
> > Hello
> >
> >
> >
> > Can anybody tell me what is the use of the HLog created per region when
a
> > region is created?
> >
> >
> >
> > Regards
> >
> > Ram
> >
> >
> >
> >
>
>
****************************************************************************
> > ***********
> > This e-mail and attachments contain confidential information from
HUAWEI,
> > which is intended only for the person or entity whose address is listed
> > above. Any use of the information contained herein in any way
(including,
> > but not limited to, total or partial disclosure, reproduction, or
> > dissemination) by persons other than the intended recipient's) is
> > prohibited. If you receive this e-mail in error, please notify the
sender
> by
> > phone or email immediately and delete it!
> >
> >
> >
>
>

Re: FW: Reg: The HLog that is created in region creation

[Ted Yu <yu...@gmail.com>]

When hbase.hstore.blockingStoreFiles is reached in one Store, updates are
blocked for this HRegion until a compaction is completed.

Regards

On Tue, Jul 5, 2011 at 12:09 AM, Ramkrishna S Vasudevan <
ramakrishnas@huawei.com> wrote:

>
> Hi all,
>
> Sorry Ted for not sending to the dev@ list.
>
> Few more queries related to splitting
>
> 1. As per the flow when compaction is happening there may be few more store
> files created due to flushing.
> 2. Suppose initially 3 Store files have been selected for compaction at the
> end of compaction process i will get 1 storefile.
> Parallely due to flushing some 3 store files were created.
>
> So after step 2 my total store files in 4.
>
> 3. Now inorder to get the midkey for splitting we iterate through all the
> store files and find the midkey from the largest store file(Correct me if
> am
> wrong).
> 4. this largest store file may be the store file that was created as part
> of
> compaction.
>
> Now how does the midkey selected will encompass all the 4 store files keys?
>
> Why i have this doubt is while splitting all the store files are moved into
> the new regions created as part of split.
>
> So all the 4 store files are now moved to both the new regions created.
>
> Pls help me in understanding this flow.
>
> Thanks in advance.
>
> Regards
> Ram
>
>
>
>
>
> ****************************************************************************
> ***********
> This e-mail and attachments contain confidential information from HUAWEI,
> which is intended only for the person or entity whose address is listed
> above. Any use of the information contained herein in any way (including,
> but not limited to, total or partial disclosure, reproduction, or
> dissemination) by persons other than the intended recipient's) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by
> phone or email immediately and delete it!
>
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Tuesday, July 05, 2011 11:01 AM
> To: ramakrishnas@huawei.com
> Cc: user@hbase.apache.org
> Subject: Re: FW: Reg: The HLog that is created in region creation
>
> Please include dev@ so that other people would be able to answer your
> question.
> >> and Daugter A2(Midkey to startkey).
> Should read (Midkey to endkey).
>
> For #1 below, you're right that splitting benefits retrieval more than
> writes.
> For #2, I guess you may have read CompactSplitThread code where compacting
> stores in a region happens before splitting.
>
> Cheers
>
> On Mon, Jul 4, 2011 at 9:26 PM, Ramkrishna S Vasudevan <
> ramakrishnas@huawei.com> wrote:
>
> > Dear Ted
> >
> > Thanks for your reply.
> >
> > I have few questions on Splitting of regions.
> >
> > I will tell my observations, may be they are not fully correct. Pls
> correct
> > me if am wrong,
> >
> > Every time compaction results in a file greater than max file size region
> > splits happen from the mid key.
> >
> > So Region A is now split into Daughter A1(Start to midkey) and Daugter
> > A2(Midkey to startkey).
> >
> > 1. Why do we do a split ?
> > I suppose it is because now a Region will now hold only a portion of the
> > full data and hence retrieval will be easy.
> >
> > 2. Is there a possibility that during first compaction a region split
> > happens, which results in 2 region creations.
> >
> > Now when new write request come to the parent region will it be
> accomodated
> > in the new daughter regions that were created.
> > If it is the case, now how will the splitkeys be handled ?
> >
> > Pls do corect me if am wrong anywhere?
> >
> > Regards
> > Ram
> >
> >
> >
> >
> >
>
> ****************************************************************************
> > ***********
> > This e-mail and attachments contain confidential information from HUAWEI,
> > which is intended only for the person or entity whose address is listed
> > above. Any use of the information contained herein in any way (including,
> > but not limited to, total or partial disclosure, reproduction, or
> > dissemination) by persons other than the intended recipient's) is
> > prohibited. If you receive this e-mail in error, please notify the sender
> > by
> > phone or email immediately and delete it!
> >
> > -----Original Message-----
> > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > Sent: Tuesday, July 05, 2011 9:38 AM
> > To: user@hbase.apache.org
> > Cc: user@hbase.apache.org
> > Subject: Re: Reg: The HLog that is created in region creation
> >
> > Not really used.
> > See hbase-4010
> >
> >
> >
> > On Jul 4, 2011, at 8:55 PM, Ramkrishna S Vasudevan <
> > ramakrishnas@huawei.com>
> > wrote:
> >
> > > Hello
> > >
> > >
> > >
> > > Can anybody tell me what is the use of the HLog created per region when
> a
> > > region is created?
> > >
> > >
> > >
> > > Regards
> > >
> > > Ram
> > >
> > >
> > >
> > >
> >
> >
>
> ****************************************************************************
> > > ***********
> > > This e-mail and attachments contain confidential information from
> HUAWEI,
> > > which is intended only for the person or entity whose address is listed
> > > above. Any use of the information contained herein in any way
> (including,
> > > but not limited to, total or partial disclosure, reproduction, or
> > > dissemination) by persons other than the intended recipient's) is
> > > prohibited. If you receive this e-mail in error, please notify the
> sender
> > by
> > > phone or email immediately and delete it!
> > >
> > >
> > >
> >
> >
>
>

Re: FW: Reg: The HLog that is created in region creation

[Ted Yu <yu...@gmail.com>]

When hbase.hstore.blockingStoreFiles is reached in one Store, updates are
blocked for this HRegion until a compaction is completed.

Regards

On Tue, Jul 5, 2011 at 12:09 AM, Ramkrishna S Vasudevan <
ramakrishnas@huawei.com> wrote:

>
> Hi all,
>
> Sorry Ted for not sending to the dev@ list.
>
> Few more queries related to splitting
>
> 1. As per the flow when compaction is happening there may be few more store
> files created due to flushing.
> 2. Suppose initially 3 Store files have been selected for compaction at the
> end of compaction process i will get 1 storefile.
> Parallely due to flushing some 3 store files were created.
>
> So after step 2 my total store files in 4.
>
> 3. Now inorder to get the midkey for splitting we iterate through all the
> store files and find the midkey from the largest store file(Correct me if
> am
> wrong).
> 4. this largest store file may be the store file that was created as part
> of
> compaction.
>
> Now how does the midkey selected will encompass all the 4 store files keys?
>
> Why i have this doubt is while splitting all the store files are moved into
> the new regions created as part of split.
>
> So all the 4 store files are now moved to both the new regions created.
>
> Pls help me in understanding this flow.
>
> Thanks in advance.
>
> Regards
> Ram
>
>
>
>
>
> ****************************************************************************
> ***********
> This e-mail and attachments contain confidential information from HUAWEI,
> which is intended only for the person or entity whose address is listed
> above. Any use of the information contained herein in any way (including,
> but not limited to, total or partial disclosure, reproduction, or
> dissemination) by persons other than the intended recipient's) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by
> phone or email immediately and delete it!
>
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Tuesday, July 05, 2011 11:01 AM
> To: ramakrishnas@huawei.com
> Cc: user@hbase.apache.org
> Subject: Re: FW: Reg: The HLog that is created in region creation
>
> Please include dev@ so that other people would be able to answer your
> question.
> >> and Daugter A2(Midkey to startkey).
> Should read (Midkey to endkey).
>
> For #1 below, you're right that splitting benefits retrieval more than
> writes.
> For #2, I guess you may have read CompactSplitThread code where compacting
> stores in a region happens before splitting.
>
> Cheers
>
> On Mon, Jul 4, 2011 at 9:26 PM, Ramkrishna S Vasudevan <
> ramakrishnas@huawei.com> wrote:
>
> > Dear Ted
> >
> > Thanks for your reply.
> >
> > I have few questions on Splitting of regions.
> >
> > I will tell my observations, may be they are not fully correct. Pls
> correct
> > me if am wrong,
> >
> > Every time compaction results in a file greater than max file size region
> > splits happen from the mid key.
> >
> > So Region A is now split into Daughter A1(Start to midkey) and Daugter
> > A2(Midkey to startkey).
> >
> > 1. Why do we do a split ?
> > I suppose it is because now a Region will now hold only a portion of the
> > full data and hence retrieval will be easy.
> >
> > 2. Is there a possibility that during first compaction a region split
> > happens, which results in 2 region creations.
> >
> > Now when new write request come to the parent region will it be
> accomodated
> > in the new daughter regions that were created.
> > If it is the case, now how will the splitkeys be handled ?
> >
> > Pls do corect me if am wrong anywhere?
> >
> > Regards
> > Ram
> >
> >
> >
> >
> >
>
> ****************************************************************************
> > ***********
> > This e-mail and attachments contain confidential information from HUAWEI,
> > which is intended only for the person or entity whose address is listed
> > above. Any use of the information contained herein in any way (including,
> > but not limited to, total or partial disclosure, reproduction, or
> > dissemination) by persons other than the intended recipient's) is
> > prohibited. If you receive this e-mail in error, please notify the sender
> > by
> > phone or email immediately and delete it!
> >
> > -----Original Message-----
> > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > Sent: Tuesday, July 05, 2011 9:38 AM
> > To: user@hbase.apache.org
> > Cc: user@hbase.apache.org
> > Subject: Re: Reg: The HLog that is created in region creation
> >
> > Not really used.
> > See hbase-4010
> >
> >
> >
> > On Jul 4, 2011, at 8:55 PM, Ramkrishna S Vasudevan <
> > ramakrishnas@huawei.com>
> > wrote:
> >
> > > Hello
> > >
> > >
> > >
> > > Can anybody tell me what is the use of the HLog created per region when
> a
> > > region is created?
> > >
> > >
> > >
> > > Regards
> > >
> > > Ram
> > >
> > >
> > >
> > >
> >
> >
>
> ****************************************************************************
> > > ***********
> > > This e-mail and attachments contain confidential information from
> HUAWEI,
> > > which is intended only for the person or entity whose address is listed
> > > above. Any use of the information contained herein in any way
> (including,
> > > but not limited to, total or partial disclosure, reproduction, or
> > > dissemination) by persons other than the intended recipient's) is
> > > prohibited. If you receive this e-mail in error, please notify the
> sender
> > by
> > > phone or email immediately and delete it!
> > >
> > >
> > >
> >
> >
>
>