You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Venkatesh <vr...@aol.com> on 2010/10/06 04:12:03 UTC

HBase map reduce job timing

 

 I've a mapreduce job that is taking too long..over an hour..Trying to see what can a tune
to to bring it down..One thing I noticed, the job is kicking off
- 500+ map tasks : 490 of them do not process any records..where as 10 of them process all the records
 (200 K each..)..Any idea why that would be?...
 
..map phase takes about couple of minutes..
..reduce phase takes the rest..

..i'll try increasing # of reduce tasks..Open to other other suggestion for tunables..

thanks for your input
venkatesh



Re: HBase map reduce job timing

Posted by Venkatesh <vr...@aol.com>.
 

 Thanks J-D
I'll hookup Ganglia (wanting but kept pushing back..) & get back
V


 

 

-----Original Message-----
From: Jean-Daniel Cryans <jd...@apache.org>
To: user@hbase.apache.org
Sent: Wed, Oct 6, 2010 12:22 pm
Subject: Re: HBase map reduce job timing


>  Also, do you think if I query using rowkey instead of hbase time stamp..it 
would not kick off that many tasks..
> since region server knows the exact locations?

I don't see how you could do that in a scalable way, unless you really
have to query a few rows (less than a million).

>
>
>
> I've a 10 node cluster each with 36 gig..I've allocated 4gig for HBase Region 
Servers..master.jsp
> reports used heap is less than half on each region server.
>

This is Java so the reported heap doesn't mean much... the garbage
collector doesn't collect aggressively since that would be awfully
inefficient.

>
>  I've close to 800 regions total..Guess it needs to kick off a jvm to see if 
data exists
> in all regions..


It does, and like you said the mappers take only a few minutes so
optimizing that part of the job is useless until you get your reducers
faster.

So regarding the speed of inserts (this seem to be the real issue if
what you said about the write buffer is true), I'd be interested in 1)
seeing your reducer's code (strip whatever you have that's business
specific) and 2) seeing some monitoring data while the job is running
(if not, get ganglia in there). Inserts could be slow for many reasons
apart from bad API usage, such as cluster misconfiguration,
sub-optimal insertion pattern (the classic being having only 1
region), etc.

J-D

 

Re: HBase map reduce job timing

Posted by Jean-Daniel Cryans <jd...@apache.org>.
>  Also, do you think if I query using rowkey instead of hbase time stamp..it would not kick off that many tasks..
> since region server knows the exact locations?

I don't see how you could do that in a scalable way, unless you really
have to query a few rows (less than a million).

>
>
>
> I've a 10 node cluster each with 36 gig..I've allocated 4gig for HBase Region Servers..master.jsp
> reports used heap is less than half on each region server.
>

This is Java so the reported heap doesn't mean much... the garbage
collector doesn't collect aggressively since that would be awfully
inefficient.

>
>  I've close to 800 regions total..Guess it needs to kick off a jvm to see if data exists
> in all regions..


It does, and like you said the mappers take only a few minutes so
optimizing that part of the job is useless until you get your reducers
faster.

So regarding the speed of inserts (this seem to be the real issue if
what you said about the write buffer is true), I'd be interested in 1)
seeing your reducer's code (strip whatever you have that's business
specific) and 2) seeing some monitoring data while the job is running
(if not, get ganglia in there). Inserts could be slow for many reasons
apart from bad API usage, such as cluster misconfiguration,
sub-optimal insertion pattern (the classic being having only 1
region), etc.

J-D

Re: HBase map reduce job timing

Posted by Venkatesh <vr...@aol.com>.
 Also, do you think if I query using rowkey instead of hbase time stamp..it would not kick off that many tasks..
since region server knows the exact locations?

thanks
venkatesh

 


 

 

-----Original Message-----
From: Venkatesh <vr...@aol.com>
To: user@hbase.apache.org
Sent: Wed, Oct 6, 2010 8:53 am
Subject: Re: HBase map reduce job timing


 Ahh ..ok..That makes sense

I've a 10 node cluster each with 36 gig..I've allocated 4gig for HBase Region Servers..master.jsp
reports used heap is less than half on each region server.


 I've close to 800 regions total..Guess it needs to kick off a jvm to see if data exists
in all regions..


 

 

-----Original Message-----
From: Jean-Daniel Cryans <jd...@apache.org>
To: user@hbase.apache.org
Sent: Tue, Oct 5, 2010 11:52 pm
Subject: Re: HBase map reduce job timing


> Regarding number of map tasks 500+, 490 of them processing nothing, do you 

have an explanation

> for that?..Wondering if its kicking off too many JVMs most doing nothing..



This would mean that throughout your regions, only a few have data in

the timestamp range you're looking for.



>

> 'top' reports less free memory (couple of gig.) though box has 36 gig total.. 

I don't quite trust

> top since cached blocks don't show up under free column even if no process is 

running..

>



You only have 1 machine?



BTW how much RAM did you give to HBase?



J-D


 
=
 

Re: HBase map reduce job timing

Posted by Venkatesh <vr...@aol.com>.
 Ahh ..ok..That makes sense

I've a 10 node cluster each with 36 gig..I've allocated 4gig for HBase Region Servers..master.jsp
reports used heap is less than half on each region server.


 I've close to 800 regions total..Guess it needs to kick off a jvm to see if data exists
in all regions..


 

 

-----Original Message-----
From: Jean-Daniel Cryans <jd...@apache.org>
To: user@hbase.apache.org
Sent: Tue, Oct 5, 2010 11:52 pm
Subject: Re: HBase map reduce job timing


> Regarding number of map tasks 500+, 490 of them processing nothing, do you 
have an explanation
> for that?..Wondering if its kicking off too many JVMs most doing nothing..

This would mean that throughout your regions, only a few have data in
the timestamp range you're looking for.

>
> 'top' reports less free memory (couple of gig.) though box has 36 gig total.. 
I don't quite trust
> top since cached blocks don't show up under free column even if no process is 
running..
>

You only have 1 machine?

BTW how much RAM did you give to HBase?

J-D

 

Re: HBase map reduce job timing

Posted by Jean-Daniel Cryans <jd...@apache.org>.
> Regarding number of map tasks 500+, 490 of them processing nothing, do you have an explanation
> for that?..Wondering if its kicking off too many JVMs most doing nothing..

This would mean that throughout your regions, only a few have data in
the timestamp range you're looking for.

>
> 'top' reports less free memory (couple of gig.) though box has 36 gig total.. I don't quite trust
> top since cached blocks don't show up under free column even if no process is running..
>

You only have 1 machine?

BTW how much RAM did you give to HBase?

J-D

Re: HBase map reduce job timing

Posted by Venkatesh <vr...@aol.com>.
 Thanks J-D
I already have 1 & 2..(write buffer & autoflush)..
I'll try 3 & 4 as well

Regarding number of map tasks 500+, 490 of them processing nothing, do you have an explanation
for that?..Wondering if its kicking off too many JVMs most doing nothing..

'top' reports less free memory (couple of gig.) though box has 36 gig total.. I don't quite trust
top since cached blocks don't show up under free column even if no process is running..

venkatesh

 


 

 

-----Original Message-----
From: Jean-Daniel Cryans <jd...@apache.org>
To: user@hbase.apache.org
Sent: Tue, Oct 5, 2010 11:30 pm
Subject: Re: HBase map reduce job timing


Ah ok, then using the write buffer should get you the speed you need
(providing that you have the hardware capacity and that you use HTable
in a efficient way).

In setup() set this to false on all 3 htables:
http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/HTable.html#setAutoFlush(boolean)

In cleanup() call this on all htables:
http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/HTable.html#flushCommits()

Also to make your maps faster you could set this to 10 or more when
you create your input format:
http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/Scan.html#setCaching(int)

J-D

On Tue, Oct 5, 2010 at 8:23 PM, Venkatesh <vr...@aol.com> wrote:
>
>  Sure..Both input & output are HBase tables
> Input (mapper phase) - scanning a HBase table for all records within time 
range (using hbase timestamps)
> Output (reduce phase) - doing a Put to 3 different HBase tables
>
>
>
> -----Original Message-----
> From: Jean-Daniel Cryans <jd...@apache.org>
> To: user@hbase.apache.org
> Sent: Tue, Oct 5, 2010 11:14 pm
> Subject: Re: HBase map reduce job timing
>
>
> It'd be more useful if we knew where that data is coming from, and
> where it's going. Are you scanning HBase and/or writing to it?
>
> J-D
>
> On Tue, Oct 5, 2010 at 8:05 PM, Venkatesh <vr...@aol.com> wrote:
>>
>>
>>
>>  Sorry..yeah..i've to do some digging to provide some data..
>> What sort of data would be helpful? Would stats reported by jobtracker.jsp
> suffice? I've pasted that in this email..
>> I can gather more jvm stats..thanks
>>
>> Status: Succeeded
>> Started at: Tue Oct 05 21:39:58 EDT 2010
>> Finished at: Tue Oct 05 22:36:43 EDT 2010
>> Finished in: 56mins, 45sec
>> Job Cleanup: Successful
>>
>>
>>
>> Kind
>> % Complete
>> Num Tasks
>> Pending
>> Running
>> Complete
>> Killed
>> Failed/Killed
>> Task Attempts
>>
>> map
>> 100.00%
>>
>>
>>
>>
>>
>> 565
>> 0
>> 0
>> 565
>> 0
>> 0 / 11
>>
>> reduce
>> 100.00%
>>
>>
>>
>>
>>
>> 20
>> 0
>> 0
>> 20
>> 0
>> 0 / 2
>>
>>
>>
>>
>>
>>
>>
>> Counter
>>
>> Map
>>
>> Reduce
>>
>> Total
>>
>>
>>
>> Job Counters
>>
>> Launched reduce tasks
>>
>> 0
>>
>> 0
>>
>> 22
>>
>>
>>
>> Rack-local map tasks
>>
>> 0
>>
>> 0
>>
>> 66
>>
>>
>>
>> Launched map tasks
>>
>> 0
>>
>> 0
>>
>> 576
>>
>>
>>
>> Data-local map tasks
>>
>> 0
>>
>> 0
>>
>> 510
>>
>>
>>
>> com.JobRecords
>>
>> REDUCE_PHASE_RECORDS
>>
>> 0
>>
>> 597,712
>>
>> 597,712
>>
>>
>>
>> MAP_PHASE_RECORDS
>>
>> 2,534,807
>>
>> 0
>>
>> 2,534,807
>>
>>
>>
>> FileSystemCounters
>>
>> FILE_BYTES_READ
>>
>> 335,845,726
>>
>> 861,146,518
>>
>> 1,196,992,244
>>
>>
>>
>> FILE_BYTES_WRITTEN
>>
>> 1,197,031,156
>>
>> 861,146,518
>>
>> 2,058,177,674
>>
>>
>>
>> Map-Reduce Framework
>>
>> Reduce input groups
>>
>> 0
>>
>> 597,712
>>
>> 597,712
>>
>>
>>
>> Combine output records
>>
>> 0
>>
>> 0
>>
>> 0
>>
>>
>>
>> Map input records
>>
>> 2,534,807
>>
>> 0
>>
>> 2,534,807
>>
>>
>>
>> Reduce shuffle bytes
>>
>> 0
>>
>> 789,145,342
>>
>> 789,145,342
>>
>>
>>
>> Reduce output records
>>
>> 0
>>
>> 0
>>
>> 0
>>
>>
>>
>> Spilled Records
>>
>> 3,522,428
>>
>> 2,534,807
>>
>> 6,057,235
>>
>>
>>
>> Map output bytes
>>
>> 851,007,170
>>
>> 0
>>
>> 851,007,170
>>
>>
>>
>> Map output records
>>
>> 2,534,807
>>
>> 0
>>
>> 2,534,807
>>
>>
>>
>> Combine input records
>>
>> 0
>>
>> 0
>>
>> 0
>>
>>
>>
>> Reduce input records
>>
>> 0
>>
>> 2,534,807
>>
>> 2,534,807
>>
>>
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Jean-Daniel Cryans <jd...@apache.org>
>> To: user@hbase.apache.org
>> Sent: Tue, Oct 5, 2010 10:53 pm
>> Subject: Re: HBase map reduce job timing
>>
>>
>> I'd love to give you tips, but you didn't provide any data about the
>> input and output of your job, the kind of hardware you're using, etc.
>> At this point any suggestion would be a stab in the dark, the best I
>> can do is pointing to the existing documentation
>> http://wiki.apache.org/hadoop/PerformanceTuning
>>
>> J-D
>>
>> On Tue, Oct 5, 2010 at 7:12 PM, Venkatesh <vr...@aol.com> wrote:
>>>
>>>
>>>
>>>  I've a mapreduce job that is taking too long..over an hour..Trying to see
>> what can a tune
>>> to to bring it down..One thing I noticed, the job is kicking off
>>> - 500+ map tasks : 490 of them do not process any records..where as 10 of
> them
>> process all the records
>>>  (200 K each..)..Any idea why that would be?...
>>>
>>> ..map phase takes about couple of minutes..
>>> ..reduce phase takes the rest..
>>>
>>> ..i'll try increasing # of reduce tasks..Open to other other suggestion for
>> tunables..
>>>
>>> thanks for your input
>>> venkatesh
>>>
>>>
>>>
>>
>>
>>
>
>
>


 

Re: HBase map reduce job timing

Posted by Jean-Daniel Cryans <jd...@apache.org>.
Ah ok, then using the write buffer should get you the speed you need
(providing that you have the hardware capacity and that you use HTable
in a efficient way).

In setup() set this to false on all 3 htables:
http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/HTable.html#setAutoFlush(boolean)

In cleanup() call this on all htables:
http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/HTable.html#flushCommits()

Also to make your maps faster you could set this to 10 or more when
you create your input format:
http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/Scan.html#setCaching(int)

J-D

On Tue, Oct 5, 2010 at 8:23 PM, Venkatesh <vr...@aol.com> wrote:
>
>  Sure..Both input & output are HBase tables
> Input (mapper phase) - scanning a HBase table for all records within time range (using hbase timestamps)
> Output (reduce phase) - doing a Put to 3 different HBase tables
>
>
>
> -----Original Message-----
> From: Jean-Daniel Cryans <jd...@apache.org>
> To: user@hbase.apache.org
> Sent: Tue, Oct 5, 2010 11:14 pm
> Subject: Re: HBase map reduce job timing
>
>
> It'd be more useful if we knew where that data is coming from, and
> where it's going. Are you scanning HBase and/or writing to it?
>
> J-D
>
> On Tue, Oct 5, 2010 at 8:05 PM, Venkatesh <vr...@aol.com> wrote:
>>
>>
>>
>>  Sorry..yeah..i've to do some digging to provide some data..
>> What sort of data would be helpful? Would stats reported by jobtracker.jsp
> suffice? I've pasted that in this email..
>> I can gather more jvm stats..thanks
>>
>> Status: Succeeded
>> Started at: Tue Oct 05 21:39:58 EDT 2010
>> Finished at: Tue Oct 05 22:36:43 EDT 2010
>> Finished in: 56mins, 45sec
>> Job Cleanup: Successful
>>
>>
>>
>> Kind
>> % Complete
>> Num Tasks
>> Pending
>> Running
>> Complete
>> Killed
>> Failed/Killed
>> Task Attempts
>>
>> map
>> 100.00%
>>
>>
>>
>>
>>
>> 565
>> 0
>> 0
>> 565
>> 0
>> 0 / 11
>>
>> reduce
>> 100.00%
>>
>>
>>
>>
>>
>> 20
>> 0
>> 0
>> 20
>> 0
>> 0 / 2
>>
>>
>>
>>
>>
>>
>>
>> Counter
>>
>> Map
>>
>> Reduce
>>
>> Total
>>
>>
>>
>> Job Counters
>>
>> Launched reduce tasks
>>
>> 0
>>
>> 0
>>
>> 22
>>
>>
>>
>> Rack-local map tasks
>>
>> 0
>>
>> 0
>>
>> 66
>>
>>
>>
>> Launched map tasks
>>
>> 0
>>
>> 0
>>
>> 576
>>
>>
>>
>> Data-local map tasks
>>
>> 0
>>
>> 0
>>
>> 510
>>
>>
>>
>> com.JobRecords
>>
>> REDUCE_PHASE_RECORDS
>>
>> 0
>>
>> 597,712
>>
>> 597,712
>>
>>
>>
>> MAP_PHASE_RECORDS
>>
>> 2,534,807
>>
>> 0
>>
>> 2,534,807
>>
>>
>>
>> FileSystemCounters
>>
>> FILE_BYTES_READ
>>
>> 335,845,726
>>
>> 861,146,518
>>
>> 1,196,992,244
>>
>>
>>
>> FILE_BYTES_WRITTEN
>>
>> 1,197,031,156
>>
>> 861,146,518
>>
>> 2,058,177,674
>>
>>
>>
>> Map-Reduce Framework
>>
>> Reduce input groups
>>
>> 0
>>
>> 597,712
>>
>> 597,712
>>
>>
>>
>> Combine output records
>>
>> 0
>>
>> 0
>>
>> 0
>>
>>
>>
>> Map input records
>>
>> 2,534,807
>>
>> 0
>>
>> 2,534,807
>>
>>
>>
>> Reduce shuffle bytes
>>
>> 0
>>
>> 789,145,342
>>
>> 789,145,342
>>
>>
>>
>> Reduce output records
>>
>> 0
>>
>> 0
>>
>> 0
>>
>>
>>
>> Spilled Records
>>
>> 3,522,428
>>
>> 2,534,807
>>
>> 6,057,235
>>
>>
>>
>> Map output bytes
>>
>> 851,007,170
>>
>> 0
>>
>> 851,007,170
>>
>>
>>
>> Map output records
>>
>> 2,534,807
>>
>> 0
>>
>> 2,534,807
>>
>>
>>
>> Combine input records
>>
>> 0
>>
>> 0
>>
>> 0
>>
>>
>>
>> Reduce input records
>>
>> 0
>>
>> 2,534,807
>>
>> 2,534,807
>>
>>
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Jean-Daniel Cryans <jd...@apache.org>
>> To: user@hbase.apache.org
>> Sent: Tue, Oct 5, 2010 10:53 pm
>> Subject: Re: HBase map reduce job timing
>>
>>
>> I'd love to give you tips, but you didn't provide any data about the
>> input and output of your job, the kind of hardware you're using, etc.
>> At this point any suggestion would be a stab in the dark, the best I
>> can do is pointing to the existing documentation
>> http://wiki.apache.org/hadoop/PerformanceTuning
>>
>> J-D
>>
>> On Tue, Oct 5, 2010 at 7:12 PM, Venkatesh <vr...@aol.com> wrote:
>>>
>>>
>>>
>>>  I've a mapreduce job that is taking too long..over an hour..Trying to see
>> what can a tune
>>> to to bring it down..One thing I noticed, the job is kicking off
>>> - 500+ map tasks : 490 of them do not process any records..where as 10 of
> them
>> process all the records
>>>  (200 K each..)..Any idea why that would be?...
>>>
>>> ..map phase takes about couple of minutes..
>>> ..reduce phase takes the rest..
>>>
>>> ..i'll try increasing # of reduce tasks..Open to other other suggestion for
>> tunables..
>>>
>>> thanks for your input
>>> venkatesh
>>>
>>>
>>>
>>
>>
>>
>
>
>

Re: HBase map reduce job timing

Posted by Venkatesh <vr...@aol.com>.
 Sure..Both input & output are HBase tables
Input (mapper phase) - scanning a HBase table for all records within time range (using hbase timestamps)
Output (reduce phase) - doing a Put to 3 different HBase tables



-----Original Message-----
From: Jean-Daniel Cryans <jd...@apache.org>
To: user@hbase.apache.org
Sent: Tue, Oct 5, 2010 11:14 pm
Subject: Re: HBase map reduce job timing


It'd be more useful if we knew where that data is coming from, and
where it's going. Are you scanning HBase and/or writing to it?

J-D

On Tue, Oct 5, 2010 at 8:05 PM, Venkatesh <vr...@aol.com> wrote:
>
>
>
>  Sorry..yeah..i've to do some digging to provide some data..
> What sort of data would be helpful? Would stats reported by jobtracker.jsp 
suffice? I've pasted that in this email..
> I can gather more jvm stats..thanks
>
> Status: Succeeded
> Started at: Tue Oct 05 21:39:58 EDT 2010
> Finished at: Tue Oct 05 22:36:43 EDT 2010
> Finished in: 56mins, 45sec
> Job Cleanup: Successful
>
>
>
> Kind
> % Complete
> Num Tasks
> Pending
> Running
> Complete
> Killed
> Failed/Killed
> Task Attempts
>
> map
> 100.00%
>
>
>
>
>
> 565
> 0
> 0
> 565
> 0
> 0 / 11
>
> reduce
> 100.00%
>
>
>
>
>
> 20
> 0
> 0
> 20
> 0
> 0 / 2
>
>
>
>
>
>
>
> Counter
>
> Map
>
> Reduce
>
> Total
>
>
>
> Job Counters
>
> Launched reduce tasks
>
> 0
>
> 0
>
> 22
>
>
>
> Rack-local map tasks
>
> 0
>
> 0
>
> 66
>
>
>
> Launched map tasks
>
> 0
>
> 0
>
> 576
>
>
>
> Data-local map tasks
>
> 0
>
> 0
>
> 510
>
>
>
> com.JobRecords
>
> REDUCE_PHASE_RECORDS
>
> 0
>
> 597,712
>
> 597,712
>
>
>
> MAP_PHASE_RECORDS
>
> 2,534,807
>
> 0
>
> 2,534,807
>
>
>
> FileSystemCounters
>
> FILE_BYTES_READ
>
> 335,845,726
>
> 861,146,518
>
> 1,196,992,244
>
>
>
> FILE_BYTES_WRITTEN
>
> 1,197,031,156
>
> 861,146,518
>
> 2,058,177,674
>
>
>
> Map-Reduce Framework
>
> Reduce input groups
>
> 0
>
> 597,712
>
> 597,712
>
>
>
> Combine output records
>
> 0
>
> 0
>
> 0
>
>
>
> Map input records
>
> 2,534,807
>
> 0
>
> 2,534,807
>
>
>
> Reduce shuffle bytes
>
> 0
>
> 789,145,342
>
> 789,145,342
>
>
>
> Reduce output records
>
> 0
>
> 0
>
> 0
>
>
>
> Spilled Records
>
> 3,522,428
>
> 2,534,807
>
> 6,057,235
>
>
>
> Map output bytes
>
> 851,007,170
>
> 0
>
> 851,007,170
>
>
>
> Map output records
>
> 2,534,807
>
> 0
>
> 2,534,807
>
>
>
> Combine input records
>
> 0
>
> 0
>
> 0
>
>
>
> Reduce input records
>
> 0
>
> 2,534,807
>
> 2,534,807
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Jean-Daniel Cryans <jd...@apache.org>
> To: user@hbase.apache.org
> Sent: Tue, Oct 5, 2010 10:53 pm
> Subject: Re: HBase map reduce job timing
>
>
> I'd love to give you tips, but you didn't provide any data about the
> input and output of your job, the kind of hardware you're using, etc.
> At this point any suggestion would be a stab in the dark, the best I
> can do is pointing to the existing documentation
> http://wiki.apache.org/hadoop/PerformanceTuning
>
> J-D
>
> On Tue, Oct 5, 2010 at 7:12 PM, Venkatesh <vr...@aol.com> wrote:
>>
>>
>>
>>  I've a mapreduce job that is taking too long..over an hour..Trying to see
> what can a tune
>> to to bring it down..One thing I noticed, the job is kicking off
>> - 500+ map tasks : 490 of them do not process any records..where as 10 of 
them
> process all the records
>>  (200 K each..)..Any idea why that would be?...
>>
>> ..map phase takes about couple of minutes..
>> ..reduce phase takes the rest..
>>
>> ..i'll try increasing # of reduce tasks..Open to other other suggestion for
> tunables..
>>
>> thanks for your input
>> venkatesh
>>
>>
>>
>
>
>

 

Re: HBase map reduce job timing

Posted by Jean-Daniel Cryans <jd...@apache.org>.
It'd be more useful if we knew where that data is coming from, and
where it's going. Are you scanning HBase and/or writing to it?

J-D

On Tue, Oct 5, 2010 at 8:05 PM, Venkatesh <vr...@aol.com> wrote:
>
>
>
>  Sorry..yeah..i've to do some digging to provide some data..
> What sort of data would be helpful? Would stats reported by jobtracker.jsp suffice? I've pasted that in this email..
> I can gather more jvm stats..thanks
>
> Status: Succeeded
> Started at: Tue Oct 05 21:39:58 EDT 2010
> Finished at: Tue Oct 05 22:36:43 EDT 2010
> Finished in: 56mins, 45sec
> Job Cleanup: Successful
>
>
>
> Kind
> % Complete
> Num Tasks
> Pending
> Running
> Complete
> Killed
> Failed/Killed
> Task Attempts
>
> map
> 100.00%
>
>
>
>
>
> 565
> 0
> 0
> 565
> 0
> 0 / 11
>
> reduce
> 100.00%
>
>
>
>
>
> 20
> 0
> 0
> 20
> 0
> 0 / 2
>
>
>
>
>
>
>
> Counter
>
> Map
>
> Reduce
>
> Total
>
>
>
> Job Counters
>
> Launched reduce tasks
>
> 0
>
> 0
>
> 22
>
>
>
> Rack-local map tasks
>
> 0
>
> 0
>
> 66
>
>
>
> Launched map tasks
>
> 0
>
> 0
>
> 576
>
>
>
> Data-local map tasks
>
> 0
>
> 0
>
> 510
>
>
>
> com.JobRecords
>
> REDUCE_PHASE_RECORDS
>
> 0
>
> 597,712
>
> 597,712
>
>
>
> MAP_PHASE_RECORDS
>
> 2,534,807
>
> 0
>
> 2,534,807
>
>
>
> FileSystemCounters
>
> FILE_BYTES_READ
>
> 335,845,726
>
> 861,146,518
>
> 1,196,992,244
>
>
>
> FILE_BYTES_WRITTEN
>
> 1,197,031,156
>
> 861,146,518
>
> 2,058,177,674
>
>
>
> Map-Reduce Framework
>
> Reduce input groups
>
> 0
>
> 597,712
>
> 597,712
>
>
>
> Combine output records
>
> 0
>
> 0
>
> 0
>
>
>
> Map input records
>
> 2,534,807
>
> 0
>
> 2,534,807
>
>
>
> Reduce shuffle bytes
>
> 0
>
> 789,145,342
>
> 789,145,342
>
>
>
> Reduce output records
>
> 0
>
> 0
>
> 0
>
>
>
> Spilled Records
>
> 3,522,428
>
> 2,534,807
>
> 6,057,235
>
>
>
> Map output bytes
>
> 851,007,170
>
> 0
>
> 851,007,170
>
>
>
> Map output records
>
> 2,534,807
>
> 0
>
> 2,534,807
>
>
>
> Combine input records
>
> 0
>
> 0
>
> 0
>
>
>
> Reduce input records
>
> 0
>
> 2,534,807
>
> 2,534,807
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Jean-Daniel Cryans <jd...@apache.org>
> To: user@hbase.apache.org
> Sent: Tue, Oct 5, 2010 10:53 pm
> Subject: Re: HBase map reduce job timing
>
>
> I'd love to give you tips, but you didn't provide any data about the
> input and output of your job, the kind of hardware you're using, etc.
> At this point any suggestion would be a stab in the dark, the best I
> can do is pointing to the existing documentation
> http://wiki.apache.org/hadoop/PerformanceTuning
>
> J-D
>
> On Tue, Oct 5, 2010 at 7:12 PM, Venkatesh <vr...@aol.com> wrote:
>>
>>
>>
>>  I've a mapreduce job that is taking too long..over an hour..Trying to see
> what can a tune
>> to to bring it down..One thing I noticed, the job is kicking off
>> - 500+ map tasks : 490 of them do not process any records..where as 10 of them
> process all the records
>>  (200 K each..)..Any idea why that would be?...
>>
>> ..map phase takes about couple of minutes..
>> ..reduce phase takes the rest..
>>
>> ..i'll try increasing # of reduce tasks..Open to other other suggestion for
> tunables..
>>
>> thanks for your input
>> venkatesh
>>
>>
>>
>
>
>

Re: HBase map reduce job timing

Posted by Venkatesh <vr...@aol.com>.
 

 Sorry..yeah..i've to do some digging to provide some data..
What sort of data would be helpful? Would stats reported by jobtracker.jsp suffice? I've pasted that in this email..
I can gather more jvm stats..thanks

Status: Succeeded
Started at: Tue Oct 05 21:39:58 EDT 2010
Finished at: Tue Oct 05 22:36:43 EDT 2010
Finished in: 56mins, 45sec
Job Cleanup: Successful



Kind
% Complete
Num Tasks
Pending
Running
Complete
Killed
Failed/Killed
Task Attempts

map
100.00%





565
0
0
565
0
0 / 11

reduce
100.00%





20
0
0
20
0
0 / 2

            

      


      
Counter
      
Map
      
Reduce
      
Total
    
            
                      
Job Counters 
                      
Launched reduce tasks
          
0
          
0
          
22
        
                
                    
Rack-local map tasks
          
0
          
0
          
66
        
                
                    
Launched map tasks
          
0
          
0
          
576
        
                
                    
Data-local map tasks
          
0
          
0
          
510
        
                
                      
com.JobRecords
                      
REDUCE_PHASE_RECORDS
          
0
          
597,712
          
597,712
        
                
                    
MAP_PHASE_RECORDS
          
2,534,807
          
0
          
2,534,807
        
                
                      
FileSystemCounters
                      
FILE_BYTES_READ
          
335,845,726
          
861,146,518
          
1,196,992,244
        
                
                    
FILE_BYTES_WRITTEN
          
1,197,031,156
          
861,146,518
          
2,058,177,674
        
                
                      
Map-Reduce Framework
                      
Reduce input groups
          
0
          
597,712
          
597,712
        
                
                    
Combine output records
          
0
          
0
          
0
        
                
                    
Map input records
          
2,534,807
          
0
          
2,534,807
        
                
                    
Reduce shuffle bytes
          
0
          
789,145,342
          
789,145,342
        
                
                    
Reduce output records
          
0
          
0
          
0
        
                
                    
Spilled Records
          
3,522,428
          
2,534,807
          
6,057,235
        
                
                    
Map output bytes
          
851,007,170
          
0
          
851,007,170
        
                
                    
Map output records
          
2,534,807
          
0
          
2,534,807
        
                
                    
Combine input records
          
0
          
0
          
0
        
                
                    
Reduce input records
          
0
          
2,534,807
          
2,534,807




 

 

-----Original Message-----
From: Jean-Daniel Cryans <jd...@apache.org>
To: user@hbase.apache.org
Sent: Tue, Oct 5, 2010 10:53 pm
Subject: Re: HBase map reduce job timing


I'd love to give you tips, but you didn't provide any data about the
input and output of your job, the kind of hardware you're using, etc.
At this point any suggestion would be a stab in the dark, the best I
can do is pointing to the existing documentation
http://wiki.apache.org/hadoop/PerformanceTuning

J-D

On Tue, Oct 5, 2010 at 7:12 PM, Venkatesh <vr...@aol.com> wrote:
>
>
>
>  I've a mapreduce job that is taking too long..over an hour..Trying to see 
what can a tune
> to to bring it down..One thing I noticed, the job is kicking off
> - 500+ map tasks : 490 of them do not process any records..where as 10 of them 
process all the records
>  (200 K each..)..Any idea why that would be?...
>
> ..map phase takes about couple of minutes..
> ..reduce phase takes the rest..
>
> ..i'll try increasing # of reduce tasks..Open to other other suggestion for 
tunables..
>
> thanks for your input
> venkatesh
>
>
>

 

Re: HBase map reduce job timing

Posted by Jean-Daniel Cryans <jd...@apache.org>.
I'd love to give you tips, but you didn't provide any data about the
input and output of your job, the kind of hardware you're using, etc.
At this point any suggestion would be a stab in the dark, the best I
can do is pointing to the existing documentation
http://wiki.apache.org/hadoop/PerformanceTuning

J-D

On Tue, Oct 5, 2010 at 7:12 PM, Venkatesh <vr...@aol.com> wrote:
>
>
>
>  I've a mapreduce job that is taking too long..over an hour..Trying to see what can a tune
> to to bring it down..One thing I noticed, the job is kicking off
> - 500+ map tasks : 490 of them do not process any records..where as 10 of them process all the records
>  (200 K each..)..Any idea why that would be?...
>
> ..map phase takes about couple of minutes..
> ..reduce phase takes the rest..
>
> ..i'll try increasing # of reduce tasks..Open to other other suggestion for tunables..
>
> thanks for your input
> venkatesh
>
>
>