You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Himanish Kushary <hi...@gmail.com> on 2011/05/24 20:31:21 UTC
HBase Transaction per second in Map-Reduce
Hi,
Could anybody please point to some article or paper which can give an
understanding of the transaction per second (both read and write) that is
supported or seen to be accomplished using HBase Map-Reduce.
We have written few HBase Map-reduces which are not giving us the
desired/expected performance .
--
Thanks & Regards
Himanish
Re: HBase Transaction per second in Map-Reduce
Posted by Himanish Kushary <hi...@gmail.com>.
To understand that whether out HBase instance is slow or is at par with
industry standards/known implementations I was looking for some article,stats
or paper for HBase TPS and performance.So the initial question :-(
It seems to be slow as per our SLA of processing the data we get.
On Tue, May 24, 2011 at 5:16 PM, Stack <sa...@gmail.com> wrote:
> Figure first what is slow before adding yet more stuff
>
>
>
> On May 24, 2011, at 14:06, Himanish Kushary <hi...@gmail.com> wrote:
>
> > Don't worry..its ok...i am going through one of those days for the last
> few
> > days :-)
> >
> > Jokes apart, when you talk about Caching, could we put something like
> > ehCache in-front of hbase integrated like a level-2 cache on top of the
> > already provided block cache ?
> >
> > - Himanish
> >
> > On Tue, May 24, 2011 at 3:19 PM, Michael Segel <
> michael_segel@hotmail.com>wrote:
> >
> >>
> >> Sorry,
> >> Its been one of those days.
> >>
> >> ----------------------------------------
> >>> From: michael_segel@hotmail.com
> >>> To: user@hbase.apache.org
> >>> Subject: RE: HBase Transaction per second in Map-Reduce
> >>> Date: Tue, 24 May 2011 14:18:28 -0500
> >>>
> >>>
> >>>
> >>> Himanish,
> >>>
> >>> Are we talking about an African or European Swallow?
> >>> (Sorry its a reference to the Monty Python movie scene where they cross
> >> the bridge after being asked 3 questions which they must answer
> correctly?
> >> [What's the forward air speed velocity of an unladen swallow?])
> >>>
> >>> The point is that you're asking a question which has no valid answer
> >> because there are so many variables that will effect the timings.
> >>>
> >>> Just to list a couple of the things that will impact your performance:
> >>> 1) Hardware
> >>> 2) Network topology
> >>> 3) Table schema.
> >>> 4) Tuning of Hadoop environment
> >>> 5) Caching? Oh wait, that could be considered Tuning.
> >>>
> >>> So while I can understand that you want to improve your performance,
> you
> >> really have to get down and look at a couple of larger issues.
> >>>
> >>> -Mike
> >>>
> >>> PS. Whatever expectations you have, you'll need to lower them. ;-)
> >>>
> >>> ----------------------------------------
> >>>> Date: Tue, 24 May 2011 14:31:21 -0400
> >>>> Subject: HBase Transaction per second in Map-Reduce
> >>>> From: himanish@gmail.com
> >>>> To: user@hbase.apache.org
> >>>>
> >>>> Hi,
> >>>>
> >>>> Could anybody please point to some article or paper which can give an
> >>>> understanding of the transaction per second (both read and write) that
> >> is
> >>>> supported or seen to be accomplished using HBase Map-Reduce.
> >>>>
> >>>> We have written few HBase Map-reduces which are not giving us the
> >>>> desired/expected performance .
> >>>>
> >>>> --
> >>>> Thanks & Regards
> >>>> Himanish
> >>>
> >>
> >>
> >
> >
> >
> > --
> > Thanks & Regards
> > Himanish
>
--
Thanks & Regards
Himanish
Re: HBase Transaction per second in Map-Reduce
Posted by Stack <sa...@gmail.com>.
Figure first what is slow before adding yet more stuff
On May 24, 2011, at 14:06, Himanish Kushary <hi...@gmail.com> wrote:
> Don't worry..its ok...i am going through one of those days for the last few
> days :-)
>
> Jokes apart, when you talk about Caching, could we put something like
> ehCache in-front of hbase integrated like a level-2 cache on top of the
> already provided block cache ?
>
> - Himanish
>
> On Tue, May 24, 2011 at 3:19 PM, Michael Segel <mi...@hotmail.com>wrote:
>
>>
>> Sorry,
>> Its been one of those days.
>>
>> ----------------------------------------
>>> From: michael_segel@hotmail.com
>>> To: user@hbase.apache.org
>>> Subject: RE: HBase Transaction per second in Map-Reduce
>>> Date: Tue, 24 May 2011 14:18:28 -0500
>>>
>>>
>>>
>>> Himanish,
>>>
>>> Are we talking about an African or European Swallow?
>>> (Sorry its a reference to the Monty Python movie scene where they cross
>> the bridge after being asked 3 questions which they must answer correctly?
>> [What's the forward air speed velocity of an unladen swallow?])
>>>
>>> The point is that you're asking a question which has no valid answer
>> because there are so many variables that will effect the timings.
>>>
>>> Just to list a couple of the things that will impact your performance:
>>> 1) Hardware
>>> 2) Network topology
>>> 3) Table schema.
>>> 4) Tuning of Hadoop environment
>>> 5) Caching? Oh wait, that could be considered Tuning.
>>>
>>> So while I can understand that you want to improve your performance, you
>> really have to get down and look at a couple of larger issues.
>>>
>>> -Mike
>>>
>>> PS. Whatever expectations you have, you'll need to lower them. ;-)
>>>
>>> ----------------------------------------
>>>> Date: Tue, 24 May 2011 14:31:21 -0400
>>>> Subject: HBase Transaction per second in Map-Reduce
>>>> From: himanish@gmail.com
>>>> To: user@hbase.apache.org
>>>>
>>>> Hi,
>>>>
>>>> Could anybody please point to some article or paper which can give an
>>>> understanding of the transaction per second (both read and write) that
>> is
>>>> supported or seen to be accomplished using HBase Map-Reduce.
>>>>
>>>> We have written few HBase Map-reduces which are not giving us the
>>>> desired/expected performance .
>>>>
>>>> --
>>>> Thanks & Regards
>>>> Himanish
>>>
>>
>>
>
>
>
> --
> Thanks & Regards
> Himanish
Re: HBase Transaction per second in Map-Reduce
Posted by Michel Segel <mi...@hotmail.com>.
You could, but you don't really need to do that.
Of course the size of your cache is tunable and based on how much memory you have.
On a side note... You said you're on CDH3, it doesn't have co-processor support, unless they snuck it in at the last minute.
What I. Think you need to do is to look at the doc in then link provided. GC tuning, good schema design will go a long way towards getting the performance you want.
Sent from a remote device. Please excuse any typos...
Mike Segel
On May 24, 2011, at 4:06 PM, Himanish Kushary <hi...@gmail.com> wrote:
> Don't worry..its ok...i am going through one of those days for the last few
> days :-)
>
> Jokes apart, when you talk about Caching, could we put something like
> ehCache in-front of hbase integrated like a level-2 cache on top of the
> already provided block cache ?
>
> - Himanish
>
> On Tue, May 24, 2011 at 3:19 PM, Michael Segel <mi...@hotmail.com>wrote:
>
>>
>> Sorry,
>> Its been one of those days.
>>
>> ----------------------------------------
>>> From: michael_segel@hotmail.com
>>> To: user@hbase.apache.org
>>> Subject: RE: HBase Transaction per second in Map-Reduce
>>> Date: Tue, 24 May 2011 14:18:28 -0500
>>>
>>>
>>>
>>> Himanish,
>>>
>>> Are we talking about an African or European Swallow?
>>> (Sorry its a reference to the Monty Python movie scene where they cross
>> the bridge after being asked 3 questions which they must answer correctly?
>> [What's the forward air speed velocity of an unladen swallow?])
>>>
>>> The point is that you're asking a question which has no valid answer
>> because there are so many variables that will effect the timings.
>>>
>>> Just to list a couple of the things that will impact your performance:
>>> 1) Hardware
>>> 2) Network topology
>>> 3) Table schema.
>>> 4) Tuning of Hadoop environment
>>> 5) Caching? Oh wait, that could be considered Tuning.
>>>
>>> So while I can understand that you want to improve your performance, you
>> really have to get down and look at a couple of larger issues.
>>>
>>> -Mike
>>>
>>> PS. Whatever expectations you have, you'll need to lower them. ;-)
>>>
>>> ----------------------------------------
>>>> Date: Tue, 24 May 2011 14:31:21 -0400
>>>> Subject: HBase Transaction per second in Map-Reduce
>>>> From: himanish@gmail.com
>>>> To: user@hbase.apache.org
>>>>
>>>> Hi,
>>>>
>>>> Could anybody please point to some article or paper which can give an
>>>> understanding of the transaction per second (both read and write) that
>> is
>>>> supported or seen to be accomplished using HBase Map-Reduce.
>>>>
>>>> We have written few HBase Map-reduces which are not giving us the
>>>> desired/expected performance .
>>>>
>>>> --
>>>> Thanks & Regards
>>>> Himanish
>>>
>>
>>
>
>
>
> --
> Thanks & Regards
> Himanish
Re: HBase Transaction per second in Map-Reduce
Posted by Himanish Kushary <hi...@gmail.com>.
Don't worry..its ok...i am going through one of those days for the last few
days :-)
Jokes apart, when you talk about Caching, could we put something like
ehCache in-front of hbase integrated like a level-2 cache on top of the
already provided block cache ?
- Himanish
On Tue, May 24, 2011 at 3:19 PM, Michael Segel <mi...@hotmail.com>wrote:
>
> Sorry,
> Its been one of those days.
>
> ----------------------------------------
> > From: michael_segel@hotmail.com
> > To: user@hbase.apache.org
> > Subject: RE: HBase Transaction per second in Map-Reduce
> > Date: Tue, 24 May 2011 14:18:28 -0500
> >
> >
> >
> > Himanish,
> >
> > Are we talking about an African or European Swallow?
> > (Sorry its a reference to the Monty Python movie scene where they cross
> the bridge after being asked 3 questions which they must answer correctly?
> [What's the forward air speed velocity of an unladen swallow?])
> >
> > The point is that you're asking a question which has no valid answer
> because there are so many variables that will effect the timings.
> >
> > Just to list a couple of the things that will impact your performance:
> > 1) Hardware
> > 2) Network topology
> > 3) Table schema.
> > 4) Tuning of Hadoop environment
> > 5) Caching? Oh wait, that could be considered Tuning.
> >
> > So while I can understand that you want to improve your performance, you
> really have to get down and look at a couple of larger issues.
> >
> > -Mike
> >
> > PS. Whatever expectations you have, you'll need to lower them. ;-)
> >
> > ----------------------------------------
> > > Date: Tue, 24 May 2011 14:31:21 -0400
> > > Subject: HBase Transaction per second in Map-Reduce
> > > From: himanish@gmail.com
> > > To: user@hbase.apache.org
> > >
> > > Hi,
> > >
> > > Could anybody please point to some article or paper which can give an
> > > understanding of the transaction per second (both read and write) that
> is
> > > supported or seen to be accomplished using HBase Map-Reduce.
> > >
> > > We have written few HBase Map-reduces which are not giving us the
> > > desired/expected performance .
> > >
> > > --
> > > Thanks & Regards
> > > Himanish
> >
>
>
--
Thanks & Regards
Himanish
RE: HBase Transaction per second in Map-Reduce
Posted by Michael Segel <mi...@hotmail.com>.
Sorry,
Its been one of those days.
----------------------------------------
> From: michael_segel@hotmail.com
> To: user@hbase.apache.org
> Subject: RE: HBase Transaction per second in Map-Reduce
> Date: Tue, 24 May 2011 14:18:28 -0500
>
>
>
> Himanish,
>
> Are we talking about an African or European Swallow?
> (Sorry its a reference to the Monty Python movie scene where they cross the bridge after being asked 3 questions which they must answer correctly? [What's the forward air speed velocity of an unladen swallow?])
>
> The point is that you're asking a question which has no valid answer because there are so many variables that will effect the timings.
>
> Just to list a couple of the things that will impact your performance:
> 1) Hardware
> 2) Network topology
> 3) Table schema.
> 4) Tuning of Hadoop environment
> 5) Caching? Oh wait, that could be considered Tuning.
>
> So while I can understand that you want to improve your performance, you really have to get down and look at a couple of larger issues.
>
> -Mike
>
> PS. Whatever expectations you have, you'll need to lower them. ;-)
>
> ----------------------------------------
> > Date: Tue, 24 May 2011 14:31:21 -0400
> > Subject: HBase Transaction per second in Map-Reduce
> > From: himanish@gmail.com
> > To: user@hbase.apache.org
> >
> > Hi,
> >
> > Could anybody please point to some article or paper which can give an
> > understanding of the transaction per second (both read and write) that is
> > supported or seen to be accomplished using HBase Map-Reduce.
> >
> > We have written few HBase Map-reduces which are not giving us the
> > desired/expected performance .
> >
> > --
> > Thanks & Regards
> > Himanish
>
RE: HBase Transaction per second in Map-Reduce
Posted by Michael Segel <mi...@hotmail.com>.
Himanish,
Are we talking about an African or European Swallow?
(Sorry its a reference to the Monty Python movie scene where they cross the bridge after being asked 3 questions which they must answer correctly? [What's the forward air speed velocity of an unladen swallow?])
The point is that you're asking a question which has no valid answer because there are so many variables that will effect the timings.
Just to list a couple of the things that will impact your performance:
1) Hardware
2) Network topology
3) Table schema.
4) Tuning of Hadoop environment
5) Caching? Oh wait, that could be considered Tuning.
So while I can understand that you want to improve your performance, you really have to get down and look at a couple of larger issues.
-Mike
PS. Whatever expectations you have, you'll need to lower them. ;-)
----------------------------------------
> Date: Tue, 24 May 2011 14:31:21 -0400
> Subject: HBase Transaction per second in Map-Reduce
> From: himanish@gmail.com
> To: user@hbase.apache.org
>
> Hi,
>
> Could anybody please point to some article or paper which can give an
> understanding of the transaction per second (both read and write) that is
> supported or seen to be accomplished using HBase Map-Reduce.
>
> We have written few HBase Map-reduces which are not giving us the
> desired/expected performance .
>
> --
> Thanks & Regards
> Himanish
Re: HBase Transaction per second in Map-Reduce
Posted by Stack <st...@duboce.net>.
See http://hbase.apache.org/book.html#performance
St.Ack
On Tue, May 24, 2011 at 11:31 AM, Himanish Kushary <hi...@gmail.com> wrote:
> Hi,
>
> Could anybody please point to some article or paper which can give an
> understanding of the transaction per second (both read and write) that is
> supported or seen to be accomplished using HBase Map-Reduce.
>
> We have written few HBase Map-reduces which are not giving us the
> desired/expected performance .
>
> --
> Thanks & Regards
> Himanish
>