You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2011/03/16 17:30:57 UTC

reduce copying of HRegionInfo's

Hi,
Please refer to 'One of the regionserver aborted, then the master shut down
itself' on user list for background.

I looked at calls to this method:
  public void assignUserRegions(List<HRegionInfo> regions, List<HServerInfo>
servers)
Here is one from AssignmentManager:
      assignUserRegions(Arrays.asList(regions.keySet().toArray(new
HRegionInfo[0])), servers);
Here is another from Master:
      this.assignmentManager.assignUserRegions(Arrays.asList(newRegions),
servers);

I propose changing the first parameter to HRegionInfo[] so that extra
copying can be avoided - considering that the number of regions in
production is high.

A bigger task is to reduce memory footprint of AssignmentManager.

I will present my further findings.

Thanks

Re: reduce copying of HRegionInfo's

Posted by Ted Yu <yu...@gmail.com>.
Patch has been uploaded to https://issues.apache.org/jira/browse/HBASE-3657

On Wed, Mar 16, 2011 at 11:12 AM, Ted Yu <yu...@gmail.com> wrote:

> It would eliminate this copy:
>
>   165       public static <T> List<T> asList(T... array) {
>   166           return new ArrayList<T>(array);
>   167       }
>   168
>
> On Wed, Mar 16, 2011 at 10:27 AM, Stack <st...@duboce.net> wrote:
>
>> Will that eliminate copying Ted?  If so, go for it (Thanks for looking
>> into this).
>> St.Ack
>>
>> On Wed, Mar 16, 2011 at 9:30 AM, Ted Yu <yu...@gmail.com> wrote:
>> > Hi,
>> > Please refer to 'One of the regionserver aborted, then the master shut
>> down
>> > itself' on user list for background.
>> >
>> > I looked at calls to this method:
>> >  public void assignUserRegions(List<HRegionInfo> regions,
>> List<HServerInfo>
>> > servers)
>> > Here is one from AssignmentManager:
>> >      assignUserRegions(Arrays.asList(regions.keySet().toArray(new
>> > HRegionInfo[0])), servers);
>> > Here is another from Master:
>> >      this.assignmentManager.assignUserRegions(Arrays.asList(newRegions),
>> > servers);
>> >
>> > I propose changing the first parameter to HRegionInfo[] so that extra
>> > copying can be avoided - considering that the number of regions in
>> > production is high.
>> >
>> > A bigger task is to reduce memory footprint of AssignmentManager.
>> >
>> > I will present my further findings.
>> >
>> > Thanks
>> >
>>
>
>

Re: reduce copying of HRegionInfo's

Posted by Ted Yu <yu...@gmail.com>.
It would eliminate this copy:

  165       public static <T> List<T> asList(T... array) {
  166           return new ArrayList<T>(array);
  167       }
  168

On Wed, Mar 16, 2011 at 10:27 AM, Stack <st...@duboce.net> wrote:

> Will that eliminate copying Ted?  If so, go for it (Thanks for looking
> into this).
> St.Ack
>
> On Wed, Mar 16, 2011 at 9:30 AM, Ted Yu <yu...@gmail.com> wrote:
> > Hi,
> > Please refer to 'One of the regionserver aborted, then the master shut
> down
> > itself' on user list for background.
> >
> > I looked at calls to this method:
> >  public void assignUserRegions(List<HRegionInfo> regions,
> List<HServerInfo>
> > servers)
> > Here is one from AssignmentManager:
> >      assignUserRegions(Arrays.asList(regions.keySet().toArray(new
> > HRegionInfo[0])), servers);
> > Here is another from Master:
> >      this.assignmentManager.assignUserRegions(Arrays.asList(newRegions),
> > servers);
> >
> > I propose changing the first parameter to HRegionInfo[] so that extra
> > copying can be avoided - considering that the number of regions in
> > production is high.
> >
> > A bigger task is to reduce memory footprint of AssignmentManager.
> >
> > I will present my further findings.
> >
> > Thanks
> >
>

Re: reduce copying of HRegionInfo's

Posted by Stack <st...@duboce.net>.
Will that eliminate copying Ted?  If so, go for it (Thanks for looking
into this).
St.Ack

On Wed, Mar 16, 2011 at 9:30 AM, Ted Yu <yu...@gmail.com> wrote:
> Hi,
> Please refer to 'One of the regionserver aborted, then the master shut down
> itself' on user list for background.
>
> I looked at calls to this method:
>  public void assignUserRegions(List<HRegionInfo> regions, List<HServerInfo>
> servers)
> Here is one from AssignmentManager:
>      assignUserRegions(Arrays.asList(regions.keySet().toArray(new
> HRegionInfo[0])), servers);
> Here is another from Master:
>      this.assignmentManager.assignUserRegions(Arrays.asList(newRegions),
> servers);
>
> I propose changing the first parameter to HRegionInfo[] so that extra
> copying can be avoided - considering that the number of regions in
> production is high.
>
> A bigger task is to reduce memory footprint of AssignmentManager.
>
> I will present my further findings.
>
> Thanks
>