You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Himanish Kushary <hi...@gmail.com> on 2012/02/20 19:35:40 UTC
Batch Get performance degrades from within Mapreduce
Hi,
We have a business scenario wherein we need to perform lot of gets for each
individual row from a Hbase table. To improve the performance we have used
the batch facilities using HTable.batch(...)
>From unit test case(run locally) ,the time taken for approx 120k gets was
between 5-7 secs . But when we run the same piece of code through
Map-Reduce (the batch get calls being made from the Mapper)
the time take is between 20 - 30 secs for far less amount of gets ( between
1200 - 2000).
Could somebody please explain why this could be happening from Map-Reduce ?
Any suggestion on how to improve this scenario is really appreciated.
--------------------------
Thanks & Regards
Himanish
Re: Batch Get performance degrades from within Mapreduce
Posted by Himanish Kushary <hi...@gmail.com>.
Hi,
We have migrated to dedicated SATA hard drives for the nodes.Now 7000 -
8000 batch GET's on a table is taking around 12-15 secs through M/R. Is
that normal ? Seems to be taking a long time.
We will have around 200K such batch GET iterations.The entire process would
take around - 200,000 X 15 secs !!!
What will be the best way to handle this scenario.The hbase table (input
table to M/R) with 200K rows are split into 8 regions after
setting MAX_FILESIZE to 10 MB. So 8 regions seems to be the best that could
be done at this moment.
Is it recommended to decrease the MAX FILESIZE for this table much further
to increase the number of splits ?
Thoughts please.
Thanks
Himanish
On Tue, Feb 21, 2012 at 6:38 PM, Himanish Kushary <hi...@gmail.com>wrote:
> Thanks for your response J-D.
>
> *You're using it in a way that doesn't make sense to me as you use a SAN*
> ,
>
> [Himanish] - Is it recommended not to use SAN while using HBase ? We were
> also planning to move to dedicated SATA hard drives on the
> production environment
>
> *don't put HBase on all nodes, have a tiny cluster... or is it just a
> testbed?*
>
> [Himanish] - You are right..it is a testbed..though even on production it
> will be small cluster , around 4-5 nodes
>
>
> *In order to do a more proper comparison I think you should run your unit
> test on one of the machines and make sure it stores the data in
> the SAN. This is one big wildcard here. Also get stats from that SAN.*
>
> [Himanish] - What kind of puzzles me is that the GETs fired from my
> laptop(unit testing) has to go over the network whereas when the M/R fires
> those GETs from the mapper,they are working on local
> data.Still the M/R GETs are performing much worse than the unit test case
>
> Also to make things a little clearer this is the pseudo code of what I am
> doing :
>
> a) GET all rows from Table 1 into RESULTs
> b) For each RESULT create a GET for fetching data from Table 2 and put it
> into a GET list
> c) Fire the batch GETs
> d) Process the RESULTs from GETs in step# (c)
>
> For M/R --------- Table 1 in Step # (a) is the input table to the M/R.Step
> b,c,d are executed within the mapper
> For Unit Testing -------- The test case takes care of all the above 4 steps
>
> Thanks
> Himanish
>
>
>
>
> On Tue, Feb 21, 2012 at 5:27 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> Something that strikes me in your answers is why have you chosen HBase
>> for this? You're using it in a way that doesn't make sense to me as
>> you use a SAN, don't put HBase on all nodes, have a tiny cluster... or
>> is it just a testbed?
>>
>> In order to do a more proper comparison I think you should run your
>> unit test on one of the machines and make sure it stores the data in
>> the SAN. This is one big wildcard here. Also get stats from that SAN.
>>
>> J-D
>>
>> On Tue, Feb 21, 2012 at 12:31 PM, Himanish Kushary <hi...@gmail.com>
>> wrote:
>> > Extremely sorry for the posts , I was also trying to provide a little
>> bit
>> > more information on our environment
>> >
>> > - You say you have 1 region server and 3 datanodes. Is there an
>> > intersection? If not, you miss out on enabling local reads and take a
>> > big performance hit although if you didn't enable it for your unit
>> > test then it's just something you might want to look at later. : The
>> region
>> > server is colocated with one of the datanodes out of the 3.
>> >
>> > - What's the machine that runs the unit test like? - Unit test is
>> running
>> > on my laptop(8 core/8 GB) through Eclipse.
>> >
>> > - How many disks per datanodes? JBOD SATA or fancier? - Datanode
>> directory
>> > are configured to point to a SAN drives
>> >
>> > - Where are the mappers running? One task tracker per datanode? Or is it
>> > per regionserver (eg 1)? - Yes, 1 TT per datanode.The server hosting the
>> > regionserver also has a TT
>> >
>> > - You say you have 8 concurrent mappers running... so I don't know if
>> > they are all on the same machine or not (see my previous question),
>> > but since you have 7 regions it means by default you can only have 7
>> > mappers running. Where's the 8th one coming from? - My mapreduce job
>> works
>> > off a table which has 8 regions.But from inside the mapper I fire
>> thousands
>> > of GET's to another different table which has 7 regions
>> >
>> > - When the MR job is running, how are the disks performing (via
>> > iostat)? Again knowing whether or not the RS is colocated with a DN
>> > would help at lot. - iostat on the regionserver during the MR shows
>> >
>> > Time: 03:27:15 PM
>> > avg-cpu: %user %nice %system %iowait %steal %idle
>> > 21.65 0.01 5.08 4.14 0.00 69.11
>> >
>> > Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
>> > sda 3.13 0.02 0.02 2176 1983
>> > sda1 0.00 0.00 0.00 8 0
>> > sda2 3.12 0.02 0.02 2167 1983
>> > sdb 52.03 2.21 0.44 289621 58030
>> > dm-0 5.41 0.02 0.02 2167 1983
>> > dm-1 0.00 0.00 0.00 0 0
>> >
>> >
>> > - Is the data set the same in the unit test and in the MR test? - The
>> data
>> > sets for the actual MR job is the same .The data set for the GETs within
>> > the mapper are much much more than from the MR ( 120000 vs 2000 GETs)
>> >
>> > -- Thanks
>> > Himanish
>> >
>> >
>> >
>> > On Tue, Feb 21, 2012 at 2:49 PM, Jean-Daniel Cryans <
>> jdcryans@apache.org>wrote:
>> >
>> >> First a side comment: if you send an email to a mailing list like this
>> >> one and didn't get any answer within a few hours, sending another one
>> >> right away usually won't help. It's just bad etiquette.
>> >>
>> >> Now I'm reading over the whole thread and things are really not that
>> >> clear to me.
>> >>
>> >> - You say you have 1 region server and 3 datanodes. Is there an
>> >> intersection? If not, you miss out on enabling local reads and take a
>> >> big performance hit although if you didn't enable it for your unit
>> >> test then it's just something you might want to look at later.
>> >>
>> >> - What's the machine that runs the unit test like?
>> >>
>> >> - How many disks per datanodes? JBOD SATA or fancier?
>> >>
>> >> - Where are the mappers running? One task tracker per datanode? Or is
>> >> it per regionserver (eg 1)?
>> >>
>> >> - You say you have 8 concurrent mappers running... so I don't know if
>> >> they are all on the same machine or not (see my previous question),
>> >> but since you have 7 regions it means by default you can only have 7
>> >> mappers running. Where's the 8th one coming from?
>> >>
>> >> - When the MR job is running, how are the disks performing (via
>> >> iostat)? Again knowing whether or not the RS is colocated with a DN
>> >> would help at lot.
>> >>
>> >> - Is the data set the same in the unit test and in the MR test?
>> >>
>> >> Thx,
>> >>
>> >> J-D
>> >>
>> >> On Mon, Feb 20, 2012 at 5:42 PM, Himanish Kushary <hi...@gmail.com>
>> >> wrote:
>> >> > Could somebody help me figure out whats the difference while running
>> >> > through map-reduce..is it just the concurrency that causing the
>> >> issue.Will
>> >> > increasing the number of region servers help ?
>> >> >
>> >> > BTW, the master is also on the same server as the regionserver.Is it
>> >> just a
>> >> > environment issue or there is some other configuration that me
>> improve
>> >> the
>> >> > read performance from within the mapper.
>> >> >
>> >> > Thanks
>> >> > Himanish
>> >>
>> >
>> >
>> >
>> > --
>> > Thanks & Regards
>> > Himanish
>>
>
>
>
> --
> Thanks & Regards
> Himanish
>
--
Thanks & Regards
Himanish
Re: Batch Get performance degrades from within Mapreduce
Posted by Himanish Kushary <hi...@gmail.com>.
Thanks for your response J-D.
*You're using it in a way that doesn't make sense to me as you use a SAN*,
[Himanish] - Is it recommended not to use SAN while using HBase ? We were
also planning to move to dedicated SATA hard drives on the
production environment
*don't put HBase on all nodes, have a tiny cluster... or is it just a
testbed?*
[Himanish] - You are right..it is a testbed..though even on production it
will be small cluster , around 4-5 nodes
*In order to do a more proper comparison I think you should run your unit
test on one of the machines and make sure it stores the data in
the SAN. This is one big wildcard here. Also get stats from that SAN.*
[Himanish] - What kind of puzzles me is that the GETs fired from my
laptop(unit testing) has to go over the network whereas when the M/R fires
those GETs from the mapper,they are working on local
data.Still the M/R GETs are performing much worse than the unit test case
Also to make things a little clearer this is the pseudo code of what I am
doing :
a) GET all rows from Table 1 into RESULTs
b) For each RESULT create a GET for fetching data from Table 2 and put it
into a GET list
c) Fire the batch GETs
d) Process the RESULTs from GETs in step# (c)
For M/R --------- Table 1 in Step # (a) is the input table to the M/R.Step
b,c,d are executed within the mapper
For Unit Testing -------- The test case takes care of all the above 4 steps
Thanks
Himanish
On Tue, Feb 21, 2012 at 5:27 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Something that strikes me in your answers is why have you chosen HBase
> for this? You're using it in a way that doesn't make sense to me as
> you use a SAN, don't put HBase on all nodes, have a tiny cluster... or
> is it just a testbed?
>
> In order to do a more proper comparison I think you should run your
> unit test on one of the machines and make sure it stores the data in
> the SAN. This is one big wildcard here. Also get stats from that SAN.
>
> J-D
>
> On Tue, Feb 21, 2012 at 12:31 PM, Himanish Kushary <hi...@gmail.com>
> wrote:
> > Extremely sorry for the posts , I was also trying to provide a little bit
> > more information on our environment
> >
> > - You say you have 1 region server and 3 datanodes. Is there an
> > intersection? If not, you miss out on enabling local reads and take a
> > big performance hit although if you didn't enable it for your unit
> > test then it's just something you might want to look at later. : The
> region
> > server is colocated with one of the datanodes out of the 3.
> >
> > - What's the machine that runs the unit test like? - Unit test is running
> > on my laptop(8 core/8 GB) through Eclipse.
> >
> > - How many disks per datanodes? JBOD SATA or fancier? - Datanode
> directory
> > are configured to point to a SAN drives
> >
> > - Where are the mappers running? One task tracker per datanode? Or is it
> > per regionserver (eg 1)? - Yes, 1 TT per datanode.The server hosting the
> > regionserver also has a TT
> >
> > - You say you have 8 concurrent mappers running... so I don't know if
> > they are all on the same machine or not (see my previous question),
> > but since you have 7 regions it means by default you can only have 7
> > mappers running. Where's the 8th one coming from? - My mapreduce job
> works
> > off a table which has 8 regions.But from inside the mapper I fire
> thousands
> > of GET's to another different table which has 7 regions
> >
> > - When the MR job is running, how are the disks performing (via
> > iostat)? Again knowing whether or not the RS is colocated with a DN
> > would help at lot. - iostat on the regionserver during the MR shows
> >
> > Time: 03:27:15 PM
> > avg-cpu: %user %nice %system %iowait %steal %idle
> > 21.65 0.01 5.08 4.14 0.00 69.11
> >
> > Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> > sda 3.13 0.02 0.02 2176 1983
> > sda1 0.00 0.00 0.00 8 0
> > sda2 3.12 0.02 0.02 2167 1983
> > sdb 52.03 2.21 0.44 289621 58030
> > dm-0 5.41 0.02 0.02 2167 1983
> > dm-1 0.00 0.00 0.00 0 0
> >
> >
> > - Is the data set the same in the unit test and in the MR test? - The
> data
> > sets for the actual MR job is the same .The data set for the GETs within
> > the mapper are much much more than from the MR ( 120000 vs 2000 GETs)
> >
> > -- Thanks
> > Himanish
> >
> >
> >
> > On Tue, Feb 21, 2012 at 2:49 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> First a side comment: if you send an email to a mailing list like this
> >> one and didn't get any answer within a few hours, sending another one
> >> right away usually won't help. It's just bad etiquette.
> >>
> >> Now I'm reading over the whole thread and things are really not that
> >> clear to me.
> >>
> >> - You say you have 1 region server and 3 datanodes. Is there an
> >> intersection? If not, you miss out on enabling local reads and take a
> >> big performance hit although if you didn't enable it for your unit
> >> test then it's just something you might want to look at later.
> >>
> >> - What's the machine that runs the unit test like?
> >>
> >> - How many disks per datanodes? JBOD SATA or fancier?
> >>
> >> - Where are the mappers running? One task tracker per datanode? Or is
> >> it per regionserver (eg 1)?
> >>
> >> - You say you have 8 concurrent mappers running... so I don't know if
> >> they are all on the same machine or not (see my previous question),
> >> but since you have 7 regions it means by default you can only have 7
> >> mappers running. Where's the 8th one coming from?
> >>
> >> - When the MR job is running, how are the disks performing (via
> >> iostat)? Again knowing whether or not the RS is colocated with a DN
> >> would help at lot.
> >>
> >> - Is the data set the same in the unit test and in the MR test?
> >>
> >> Thx,
> >>
> >> J-D
> >>
> >> On Mon, Feb 20, 2012 at 5:42 PM, Himanish Kushary <hi...@gmail.com>
> >> wrote:
> >> > Could somebody help me figure out whats the difference while running
> >> > through map-reduce..is it just the concurrency that causing the
> >> issue.Will
> >> > increasing the number of region servers help ?
> >> >
> >> > BTW, the master is also on the same server as the regionserver.Is it
> >> just a
> >> > environment issue or there is some other configuration that me improve
> >> the
> >> > read performance from within the mapper.
> >> >
> >> > Thanks
> >> > Himanish
> >>
> >
> >
> >
> > --
> > Thanks & Regards
> > Himanish
>
--
Thanks & Regards
Himanish
Re: Batch Get performance degrades from within Mapreduce
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Something that strikes me in your answers is why have you chosen HBase
for this? You're using it in a way that doesn't make sense to me as
you use a SAN, don't put HBase on all nodes, have a tiny cluster... or
is it just a testbed?
In order to do a more proper comparison I think you should run your
unit test on one of the machines and make sure it stores the data in
the SAN. This is one big wildcard here. Also get stats from that SAN.
J-D
On Tue, Feb 21, 2012 at 12:31 PM, Himanish Kushary <hi...@gmail.com> wrote:
> Extremely sorry for the posts , I was also trying to provide a little bit
> more information on our environment
>
> - You say you have 1 region server and 3 datanodes. Is there an
> intersection? If not, you miss out on enabling local reads and take a
> big performance hit although if you didn't enable it for your unit
> test then it's just something you might want to look at later. : The region
> server is colocated with one of the datanodes out of the 3.
>
> - What's the machine that runs the unit test like? - Unit test is running
> on my laptop(8 core/8 GB) through Eclipse.
>
> - How many disks per datanodes? JBOD SATA or fancier? - Datanode directory
> are configured to point to a SAN drives
>
> - Where are the mappers running? One task tracker per datanode? Or is it
> per regionserver (eg 1)? - Yes, 1 TT per datanode.The server hosting the
> regionserver also has a TT
>
> - You say you have 8 concurrent mappers running... so I don't know if
> they are all on the same machine or not (see my previous question),
> but since you have 7 regions it means by default you can only have 7
> mappers running. Where's the 8th one coming from? - My mapreduce job works
> off a table which has 8 regions.But from inside the mapper I fire thousands
> of GET's to another different table which has 7 regions
>
> - When the MR job is running, how are the disks performing (via
> iostat)? Again knowing whether or not the RS is colocated with a DN
> would help at lot. - iostat on the regionserver during the MR shows
>
> Time: 03:27:15 PM
> avg-cpu: %user %nice %system %iowait %steal %idle
> 21.65 0.01 5.08 4.14 0.00 69.11
>
> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> sda 3.13 0.02 0.02 2176 1983
> sda1 0.00 0.00 0.00 8 0
> sda2 3.12 0.02 0.02 2167 1983
> sdb 52.03 2.21 0.44 289621 58030
> dm-0 5.41 0.02 0.02 2167 1983
> dm-1 0.00 0.00 0.00 0 0
>
>
> - Is the data set the same in the unit test and in the MR test? - The data
> sets for the actual MR job is the same .The data set for the GETs within
> the mapper are much much more than from the MR ( 120000 vs 2000 GETs)
>
> -- Thanks
> Himanish
>
>
>
> On Tue, Feb 21, 2012 at 2:49 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> First a side comment: if you send an email to a mailing list like this
>> one and didn't get any answer within a few hours, sending another one
>> right away usually won't help. It's just bad etiquette.
>>
>> Now I'm reading over the whole thread and things are really not that
>> clear to me.
>>
>> - You say you have 1 region server and 3 datanodes. Is there an
>> intersection? If not, you miss out on enabling local reads and take a
>> big performance hit although if you didn't enable it for your unit
>> test then it's just something you might want to look at later.
>>
>> - What's the machine that runs the unit test like?
>>
>> - How many disks per datanodes? JBOD SATA or fancier?
>>
>> - Where are the mappers running? One task tracker per datanode? Or is
>> it per regionserver (eg 1)?
>>
>> - You say you have 8 concurrent mappers running... so I don't know if
>> they are all on the same machine or not (see my previous question),
>> but since you have 7 regions it means by default you can only have 7
>> mappers running. Where's the 8th one coming from?
>>
>> - When the MR job is running, how are the disks performing (via
>> iostat)? Again knowing whether or not the RS is colocated with a DN
>> would help at lot.
>>
>> - Is the data set the same in the unit test and in the MR test?
>>
>> Thx,
>>
>> J-D
>>
>> On Mon, Feb 20, 2012 at 5:42 PM, Himanish Kushary <hi...@gmail.com>
>> wrote:
>> > Could somebody help me figure out whats the difference while running
>> > through map-reduce..is it just the concurrency that causing the
>> issue.Will
>> > increasing the number of region servers help ?
>> >
>> > BTW, the master is also on the same server as the regionserver.Is it
>> just a
>> > environment issue or there is some other configuration that me improve
>> the
>> > read performance from within the mapper.
>> >
>> > Thanks
>> > Himanish
>>
>
>
>
> --
> Thanks & Regards
> Himanish
Re: Batch Get performance degrades from within Mapreduce
Posted by Himanish Kushary <hi...@gmail.com>.
Extremely sorry for the posts , I was also trying to provide a little bit
more information on our environment
- You say you have 1 region server and 3 datanodes. Is there an
intersection? If not, you miss out on enabling local reads and take a
big performance hit although if you didn't enable it for your unit
test then it's just something you might want to look at later. : The region
server is colocated with one of the datanodes out of the 3.
- What's the machine that runs the unit test like? - Unit test is running
on my laptop(8 core/8 GB) through Eclipse.
- How many disks per datanodes? JBOD SATA or fancier? - Datanode directory
are configured to point to a SAN drives
- Where are the mappers running? One task tracker per datanode? Or is it
per regionserver (eg 1)? - Yes, 1 TT per datanode.The server hosting the
regionserver also has a TT
- You say you have 8 concurrent mappers running... so I don't know if
they are all on the same machine or not (see my previous question),
but since you have 7 regions it means by default you can only have 7
mappers running. Where's the 8th one coming from? - My mapreduce job works
off a table which has 8 regions.But from inside the mapper I fire thousands
of GET's to another different table which has 7 regions
- When the MR job is running, how are the disks performing (via
iostat)? Again knowing whether or not the RS is colocated with a DN
would help at lot. - iostat on the regionserver during the MR shows
Time: 03:27:15 PM
avg-cpu: %user %nice %system %iowait %steal %idle
21.65 0.01 5.08 4.14 0.00 69.11
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 3.13 0.02 0.02 2176 1983
sda1 0.00 0.00 0.00 8 0
sda2 3.12 0.02 0.02 2167 1983
sdb 52.03 2.21 0.44 289621 58030
dm-0 5.41 0.02 0.02 2167 1983
dm-1 0.00 0.00 0.00 0 0
- Is the data set the same in the unit test and in the MR test? - The data
sets for the actual MR job is the same .The data set for the GETs within
the mapper are much much more than from the MR ( 120000 vs 2000 GETs)
-- Thanks
Himanish
On Tue, Feb 21, 2012 at 2:49 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> First a side comment: if you send an email to a mailing list like this
> one and didn't get any answer within a few hours, sending another one
> right away usually won't help. It's just bad etiquette.
>
> Now I'm reading over the whole thread and things are really not that
> clear to me.
>
> - You say you have 1 region server and 3 datanodes. Is there an
> intersection? If not, you miss out on enabling local reads and take a
> big performance hit although if you didn't enable it for your unit
> test then it's just something you might want to look at later.
>
> - What's the machine that runs the unit test like?
>
> - How many disks per datanodes? JBOD SATA or fancier?
>
> - Where are the mappers running? One task tracker per datanode? Or is
> it per regionserver (eg 1)?
>
> - You say you have 8 concurrent mappers running... so I don't know if
> they are all on the same machine or not (see my previous question),
> but since you have 7 regions it means by default you can only have 7
> mappers running. Where's the 8th one coming from?
>
> - When the MR job is running, how are the disks performing (via
> iostat)? Again knowing whether or not the RS is colocated with a DN
> would help at lot.
>
> - Is the data set the same in the unit test and in the MR test?
>
> Thx,
>
> J-D
>
> On Mon, Feb 20, 2012 at 5:42 PM, Himanish Kushary <hi...@gmail.com>
> wrote:
> > Could somebody help me figure out whats the difference while running
> > through map-reduce..is it just the concurrency that causing the
> issue.Will
> > increasing the number of region servers help ?
> >
> > BTW, the master is also on the same server as the regionserver.Is it
> just a
> > environment issue or there is some other configuration that me improve
> the
> > read performance from within the mapper.
> >
> > Thanks
> > Himanish
>
--
Thanks & Regards
Himanish
Re: Batch Get performance degrades from within Mapreduce
Posted by Jean-Daniel Cryans <jd...@apache.org>.
First a side comment: if you send an email to a mailing list like this
one and didn't get any answer within a few hours, sending another one
right away usually won't help. It's just bad etiquette.
Now I'm reading over the whole thread and things are really not that
clear to me.
- You say you have 1 region server and 3 datanodes. Is there an
intersection? If not, you miss out on enabling local reads and take a
big performance hit although if you didn't enable it for your unit
test then it's just something you might want to look at later.
- What's the machine that runs the unit test like?
- How many disks per datanodes? JBOD SATA or fancier?
- Where are the mappers running? One task tracker per datanode? Or is
it per regionserver (eg 1)?
- You say you have 8 concurrent mappers running... so I don't know if
they are all on the same machine or not (see my previous question),
but since you have 7 regions it means by default you can only have 7
mappers running. Where's the 8th one coming from?
- When the MR job is running, how are the disks performing (via
iostat)? Again knowing whether or not the RS is colocated with a DN
would help at lot.
- Is the data set the same in the unit test and in the MR test?
Thx,
J-D
On Mon, Feb 20, 2012 at 5:42 PM, Himanish Kushary <hi...@gmail.com> wrote:
> Could somebody help me figure out whats the difference while running
> through map-reduce..is it just the concurrency that causing the issue.Will
> increasing the number of region servers help ?
>
> BTW, the master is also on the same server as the regionserver.Is it just a
> environment issue or there is some other configuration that me improve the
> read performance from within the mapper.
>
> Thanks
> Himanish
Re: Batch Get performance degrades from within Mapreduce
Posted by Himanish Kushary <hi...@gmail.com>.
Could somebody help me figure out whats the difference while running
through map-reduce..is it just the concurrency that causing the issue.Will
increasing the number of region servers help ?
BTW, the master is also on the same server as the regionserver.Is it just a
environment issue or there is some other configuration that me improve the
read performance from within the mapper.
Thanks
Himanish
On Mon, Feb 20, 2012 at 4:54 PM, Himanish Kushary <hi...@gmail.com>wrote:
> Actually 1200-2000 GETs at a time from inside the mapper.From unit test
> case 120,000 GETs at a time. From Mapper it takes 20-30 secs whereas from
> test case ( the much bigger GETs call) takes just 5-6 secs.
>
> The rows are pretty small(one LONG value)..7 regions in the single region
> server for the concerned table
>
> When the map-reduce is running ..On the regionserver sometimes the free
> memory ( using free -m ) coming down to 100 MB , CPU usage (using top
> command ) for hbase going between 300-400%
>
> Here is the regionserver configuration :
>
> RAM - 20G for the entire box with 8G heap size for the
> reionserver(Swapiness = 0)
> CPU - 8 Core
> RegionServer handler count = 50
> Compression - Enabled (Snappy)
> Table File Max Size = 3G
> Block Cache,Memstore left at default settings
> No of datanodes = 3
>
>
> My main concern is - Why does the same batch gets perform much much better
> when run through test cases compared with running it through Map-Reduce.It
> may make sense probably only due to the concurrency issue.
>
> - Thank
> Himanish
>
>
> On Mon, Feb 20, 2012 at 4:37 PM, Stack <st...@duboce.net> wrote:
>
>> On Mon, Feb 20, 2012 at 1:12 PM, Himanish Kushary <hi...@gmail.com>
>> wrote:
>> > Where is the time being spent? In the server, in the mapper? - The most
>> > time is spent in calling Htable.batch(...) inside the mapper
>> >
>>
>> So, 100 Gets at a time?
>>
>>
>> > Why are you having scanner timeouts if you are doing big batch Gets? -
>> We
>> > are getting scanner timeout from the original Scan which serves the
>> input
>> > records to the mapper.The scanner caching is set to 100 .
>> >
>> > I think because the mapper is taking too
>> > long(because of the batch Gets inside it) to process initial 100
>> records ,
>> > the next batch of scanned records throws the exception
>> >
>> >
>>
>>
>> How big are these 100 rows?
>>
>> How many regions on this single RegionServer?
>>
>>
>> > Also, could it be happening due to concurrency ? I am currently on a
>> single
>> > region-server. When i run the test case the batch Gets happen
>> sequentially
>> > whereas from the map-reduce the batch Gets happen concurrently on the
>> same
>> > region server. Could this be the reason that during map-reduce the
>> > performance degrades due to thrashing on the same region server ?
>> Thoughts ?
>> >
>>
>> A single regionserver? How many datanodes?
>>
>> Whats it look like on the machine running the regionserver? Is it
>> working hard? Seems odd that 100 gets would take longer than a a
>> minute to complete.
>>
>> You've checked out the performance section of the reference guide?
>>
>> St.Ack
>>
>
>
>
> --
> Thanks & Regards
> Himanish
>
--
Thanks & Regards
Himanish
Re: Batch Get performance degrades from within Mapreduce
Posted by Himanish Kushary <hi...@gmail.com>.
Actually 1200-2000 GETs at a time from inside the mapper.From unit test
case 120,000 GETs at a time. From Mapper it takes 20-30 secs whereas from
test case ( the much bigger GETs call) takes just 5-6 secs.
The rows are pretty small(one LONG value)..7 regions in the single region
server for the concerned table
When the map-reduce is running ..On the regionserver sometimes the free
memory ( using free -m ) coming down to 100 MB , CPU usage (using top
command ) for hbase going between 300-400%
Here is the regionserver configuration :
RAM - 20G for the entire box with 8G heap size for the
reionserver(Swapiness = 0)
CPU - 8 Core
RegionServer handler count = 50
Compression - Enabled (Snappy)
Table File Max Size = 3G
Block Cache,Memstore left at default settings
No of datanodes = 3
My main concern is - Why does the same batch gets perform much much better
when run through test cases compared with running it through Map-Reduce.It
may make sense probably only due to the concurrency issue.
- Thank
Himanish
On Mon, Feb 20, 2012 at 4:37 PM, Stack <st...@duboce.net> wrote:
> On Mon, Feb 20, 2012 at 1:12 PM, Himanish Kushary <hi...@gmail.com>
> wrote:
> > Where is the time being spent? In the server, in the mapper? - The most
> > time is spent in calling Htable.batch(...) inside the mapper
> >
>
> So, 100 Gets at a time?
>
>
> > Why are you having scanner timeouts if you are doing big batch Gets? - We
> > are getting scanner timeout from the original Scan which serves the input
> > records to the mapper.The scanner caching is set to 100 .
> >
> > I think because the mapper is taking too
> > long(because of the batch Gets inside it) to process initial 100 records
> ,
> > the next batch of scanned records throws the exception
> >
> >
>
>
> How big are these 100 rows?
>
> How many regions on this single RegionServer?
>
>
> > Also, could it be happening due to concurrency ? I am currently on a
> single
> > region-server. When i run the test case the batch Gets happen
> sequentially
> > whereas from the map-reduce the batch Gets happen concurrently on the
> same
> > region server. Could this be the reason that during map-reduce the
> > performance degrades due to thrashing on the same region server ?
> Thoughts ?
> >
>
> A single regionserver? How many datanodes?
>
> Whats it look like on the machine running the regionserver? Is it
> working hard? Seems odd that 100 gets would take longer than a a
> minute to complete.
>
> You've checked out the performance section of the reference guide?
>
> St.Ack
>
--
Thanks & Regards
Himanish
Re: Batch Get performance degrades from within Mapreduce
Posted by Stack <st...@duboce.net>.
On Mon, Feb 20, 2012 at 1:12 PM, Himanish Kushary <hi...@gmail.com> wrote:
> Where is the time being spent? In the server, in the mapper? - The most
> time is spent in calling Htable.batch(...) inside the mapper
>
So, 100 Gets at a time?
> Why are you having scanner timeouts if you are doing big batch Gets? - We
> are getting scanner timeout from the original Scan which serves the input
> records to the mapper.The scanner caching is set to 100 .
>
> I think because the mapper is taking too
> long(because of the batch Gets inside it) to process initial 100 records ,
> the next batch of scanned records throws the exception
>
>
How big are these 100 rows?
How many regions on this single RegionServer?
> Also, could it be happening due to concurrency ? I am currently on a single
> region-server. When i run the test case the batch Gets happen sequentially
> whereas from the map-reduce the batch Gets happen concurrently on the same
> region server. Could this be the reason that during map-reduce the
> performance degrades due to thrashing on the same region server ? Thoughts ?
>
A single regionserver? How many datanodes?
Whats it look like on the machine running the regionserver? Is it
working hard? Seems odd that 100 gets would take longer than a a
minute to complete.
You've checked out the performance section of the reference guide?
St.Ack
Re: Batch Get performance degrades from within Mapreduce
Posted by Himanish Kushary <hi...@gmail.com>.
Thanks for the reply.Here are the info :
What is your concurrency like (How many concurrent mappers?). - We have 8
concurrent mappers running
Where is the time being spent? In the server, in the mapper? - The most
time is spent in calling Htable.batch(...) inside the mapper
Why are you having scanner timeouts if you are doing big batch Gets? - We
are getting scanner timeout from the original Scan which serves the input
records to the mapper.The scanner caching is set to 100 .
I think because the mapper is taking too
long(because of the batch Gets inside it) to process initial 100 records ,
the next batch of scanned records throws the exception
Also, could it be happening due to concurrency ? I am currently on a single
region-server. When i run the test case the batch Gets happen sequentially
whereas from the map-reduce the batch Gets happen concurrently on the same
region server. Could this be the reason that during map-reduce the
performance degrades due to thrashing on the same region server ? Thoughts ?
- Thanks
Himanish
On Mon, Feb 20, 2012 at 3:39 PM, Stack <st...@duboce.net> wrote:
> On Mon, Feb 20, 2012 at 12:04 PM, Himanish Kushary <hi...@gmail.com>
> wrote:
> > Also to add , from the map-reduce we have started seeing
> >
> > org.apache.hadoop.hbase.client.ScannerTimeoutException: 360388ms
> > passed since the last invocation, timeout is currently set to 300000
> >
> > due to the extremely high time spent on firing the batch Gets
> >
>
> What is your concurrency like (How many concurrent mappers?). Where
> is the time being spent? In the server, in the mapper? Why are you
> having scanner timeouts if you are doing big batch Gets?
>
> St.Ack
>
--
Thanks & Regards
Himanish
Re: Batch Get performance degrades from within Mapreduce
Posted by Stack <st...@duboce.net>.
On Mon, Feb 20, 2012 at 12:04 PM, Himanish Kushary <hi...@gmail.com> wrote:
> Also to add , from the map-reduce we have started seeing
>
> org.apache.hadoop.hbase.client.ScannerTimeoutException: 360388ms
> passed since the last invocation, timeout is currently set to 300000
>
> due to the extremely high time spent on firing the batch Gets
>
What is your concurrency like (How many concurrent mappers?). Where
is the time being spent? In the server, in the mapper? Why are you
having scanner timeouts if you are doing big batch Gets?
St.Ack
Re: Batch Get performance degrades from within Mapreduce
Posted by Himanish Kushary <hi...@gmail.com>.
Also to add , from the map-reduce we have started seeing
org.apache.hadoop.hbase.client.ScannerTimeoutException: 360388ms
passed since the last invocation, timeout is currently set to 300000
due to the extremely high time spent on firing the batch Gets
- Himanish
On Mon, Feb 20, 2012 at 2:50 PM, Himanish Kushary <hi...@gmail.com>wrote:
> Hi,
>
> We are on CDH3u3 .
>
> - Himanish
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Hi,
>
> We have a business scenario wherein we need to perform lot of gets for
> each individual row from a Hbase table. To improve the performance we have
> used the batch facilities using HTable.batch(...)
>
> From unit test case(run locally) ,the time taken for approx 120k gets was
> between 5-7 secs . But when we run the same piece of code through
> Map-Reduce (the batch get calls being made from the Mapper)
> the time take is between 20 - 30 secs for far less amount of gets (
> between 1200 - 2000).
>
> Could somebody please explain why this could be happening from Map-Reduce
> ? Any suggestion on how to improve this scenario is really appreciated.
>
> --------------------------
> Thanks & Regards
> Himanish
>
>
>
> --
> Thanks & Regards
> Himanish
>
--
Thanks & Regards
Himanish
Fwd: Batch Get performance degrades from within Mapreduce
Posted by Himanish Kushary <hi...@gmail.com>.
Hi,
We are on CDH3u3 .
- Himanish
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Hi,
We have a business scenario wherein we need to perform lot of gets for each
individual row from a Hbase table. To improve the performance we have used
the batch facilities using HTable.batch(...)
>From unit test case(run locally) ,the time taken for approx 120k gets was
between 5-7 secs . But when we run the same piece of code through
Map-Reduce (the batch get calls being made from the Mapper)
the time take is between 20 - 30 secs for far less amount of gets ( between
1200 - 2000).
Could somebody please explain why this could be happening from Map-Reduce ?
Any suggestion on how to improve this scenario is really appreciated.
--------------------------
Thanks & Regards
Himanish
--
Thanks & Regards
Himanish