You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by praveenesh kumar <pr...@gmail.com> on 2012/01/25 11:04:14 UTC
Reduce > copy at 0.00 MB/s
Hey,
Can anyone explain me what is reduce > copy phase in the reducer section ?
The (K,List(V)), is passed to the reducer. Is reduce > copy representing
copying of K,List(V) on the reducer from all mappers ?
I am monitoring my jobs on the cluster, using Jobtracker url.
I am seeing for most of my reducing jobs, something like this :
task_201201250352_0001_r_000000 31.05% reduce > copy (395 of 424 at 0.00
MB/s) > 25-Jan-2012 03:54:06
task_201201250352_0001_r_000001 30.73% reduce > copy (391 of 424 at 0.00
MB/s) > 25-Jan-2012 03:54:06
task_201201250352_0001_r_000002 30.89% reduce > copy (393 of 424 at 0.00
MB/s) > 25-Jan-2012 03:54:06
............................
Can anyone explain me why the speed is 0.00 MB/s. Job is running fine.
Is it because this reduce > copy is happening on the same machine.
Thanks,
Praveenesh
RE: Reduce > copy at 0.00 MB/s
Posted by sathyavageeswaran <sa...@morisonmenon.com>.
Can somebody help me to unsubscribe.
Even after unsubscribing I continue to get mails.
-----Original Message-----
From: hadoop hive [mailto:hadoophive@gmail.com]
Sent: 02 February 2012 12:29
To: common-user@hadoop.apache.org
Subject: Re: Reduce > copy at 0.00 MB/s
Hey ,
Can any1 help me with this, i have increases the reduce slowstart to .25
but its still hangs after copy .
tell me what else i can change it to make it working fine.
regards
Vikas Srivastava
On Wed, Jan 25, 2012 at 7:45 PM, praveenesh kumar
<pr...@gmail.com>wrote:
> Yeah , I am doing it, currently its on 20 %, I guess I have to raise it
> more.
> Funny thing is, its still happening after map is 100% completed.
> when map is completed, it should not wait, right. But I see it still give
> same message, for some time.
>
> Thanks,
> Praveenesh
>
> On Wed, Jan 25, 2012 at 7:29 PM, Harsh J <ha...@cloudera.com> wrote:
>
> > The copy phase fetches the map outputs. It may hang for a while if
> > there are no newly completed map outputs to fetch yet.
> >
> > You can raise your reducers' slowstart value to have it not spend so
> > many cycles waiting but rather start at 80-90% of map completions,
> > instead of default 5%. This helps your MR performance overall, if you
> > run multiple jobs at a time, as the reduce slots aren't wasted.
> >
> > On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar <pr...@gmail.com>
> > wrote:
> > > Hey,
> > >
> > > Can anyone explain me what is reduce > copy phase in the reducer
> section
> > ?
> > > The (K,List(V)), is passed to the reducer. Is reduce > copy
> representing
> > > copying of K,List(V) on the reducer from all mappers ?
> > >
> > > I am monitoring my jobs on the cluster, using Jobtracker url.
> > > I am seeing for most of my reducing jobs, something like this :
> > >
> > > task_201201250352_0001_r_000000 31.05% reduce > copy (395 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > > task_201201250352_0001_r_000001 30.73% reduce > copy (391 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > > task_201201250352_0001_r_000002 30.89% reduce > copy (393 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > >
> > > ............................
> > >
> > > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine.
> > > Is it because this reduce > copy is happening on the same machine.
> > >
> > > Thanks,
> > > Praveenesh
> >
> >
> >
> > --
> > Harsh J
> > Customer Ops. Engineer, Cloudera
> >
>
-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1913 / Virus Database: 2112/4780 - Release Date: 02/01/12
Re: Reduce > copy at 0.00 MB/s
Posted by hadoop hive <ha...@gmail.com>.
Hey ,
Can any1 help me with this, i have increases the reduce slowstart to .25
but its still hangs after copy .
tell me what else i can change it to make it working fine.
regards
Vikas Srivastava
On Wed, Jan 25, 2012 at 7:45 PM, praveenesh kumar <pr...@gmail.com>wrote:
> Yeah , I am doing it, currently its on 20 %, I guess I have to raise it
> more.
> Funny thing is, its still happening after map is 100% completed.
> when map is completed, it should not wait, right. But I see it still give
> same message, for some time.
>
> Thanks,
> Praveenesh
>
> On Wed, Jan 25, 2012 at 7:29 PM, Harsh J <ha...@cloudera.com> wrote:
>
> > The copy phase fetches the map outputs. It may hang for a while if
> > there are no newly completed map outputs to fetch yet.
> >
> > You can raise your reducers' slowstart value to have it not spend so
> > many cycles waiting but rather start at 80-90% of map completions,
> > instead of default 5%. This helps your MR performance overall, if you
> > run multiple jobs at a time, as the reduce slots aren't wasted.
> >
> > On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar <pr...@gmail.com>
> > wrote:
> > > Hey,
> > >
> > > Can anyone explain me what is reduce > copy phase in the reducer
> section
> > ?
> > > The (K,List(V)), is passed to the reducer. Is reduce > copy
> representing
> > > copying of K,List(V) on the reducer from all mappers ?
> > >
> > > I am monitoring my jobs on the cluster, using Jobtracker url.
> > > I am seeing for most of my reducing jobs, something like this :
> > >
> > > task_201201250352_0001_r_000000 31.05% reduce > copy (395 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > > task_201201250352_0001_r_000001 30.73% reduce > copy (391 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > > task_201201250352_0001_r_000002 30.89% reduce > copy (393 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > >
> > > ............................
> > >
> > > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine.
> > > Is it because this reduce > copy is happening on the same machine.
> > >
> > > Thanks,
> > > Praveenesh
> >
> >
> >
> > --
> > Harsh J
> > Customer Ops. Engineer, Cloudera
> >
>
Re: Reduce > copy at 0.00 MB/s
Posted by praveenesh kumar <pr...@gmail.com>.
Yeah , I am doing it, currently its on 20 %, I guess I have to raise it
more.
Funny thing is, its still happening after map is 100% completed.
when map is completed, it should not wait, right. But I see it still give
same message, for some time.
Thanks,
Praveenesh
On Wed, Jan 25, 2012 at 7:29 PM, Harsh J <ha...@cloudera.com> wrote:
> The copy phase fetches the map outputs. It may hang for a while if
> there are no newly completed map outputs to fetch yet.
>
> You can raise your reducers' slowstart value to have it not spend so
> many cycles waiting but rather start at 80-90% of map completions,
> instead of default 5%. This helps your MR performance overall, if you
> run multiple jobs at a time, as the reduce slots aren't wasted.
>
> On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar <pr...@gmail.com>
> wrote:
> > Hey,
> >
> > Can anyone explain me what is reduce > copy phase in the reducer section
> ?
> > The (K,List(V)), is passed to the reducer. Is reduce > copy representing
> > copying of K,List(V) on the reducer from all mappers ?
> >
> > I am monitoring my jobs on the cluster, using Jobtracker url.
> > I am seeing for most of my reducing jobs, something like this :
> >
> > task_201201250352_0001_r_000000 31.05% reduce > copy (395 of 424 at
> 0.00
> > MB/s) > 25-Jan-2012 03:54:06
> >
> > task_201201250352_0001_r_000001 30.73% reduce > copy (391 of 424 at
> 0.00
> > MB/s) > 25-Jan-2012 03:54:06
> >
> > task_201201250352_0001_r_000002 30.89% reduce > copy (393 of 424 at
> 0.00
> > MB/s) > 25-Jan-2012 03:54:06
> >
> >
> > ............................
> >
> > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine.
> > Is it because this reduce > copy is happening on the same machine.
> >
> > Thanks,
> > Praveenesh
>
>
>
> --
> Harsh J
> Customer Ops. Engineer, Cloudera
>
Re: Reduce > copy at 0.00 MB/s
Posted by Harsh J <ha...@cloudera.com>.
The copy phase fetches the map outputs. It may hang for a while if
there are no newly completed map outputs to fetch yet.
You can raise your reducers' slowstart value to have it not spend so
many cycles waiting but rather start at 80-90% of map completions,
instead of default 5%. This helps your MR performance overall, if you
run multiple jobs at a time, as the reduce slots aren't wasted.
On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar <pr...@gmail.com> wrote:
> Hey,
>
> Can anyone explain me what is reduce > copy phase in the reducer section ?
> The (K,List(V)), is passed to the reducer. Is reduce > copy representing
> copying of K,List(V) on the reducer from all mappers ?
>
> I am monitoring my jobs on the cluster, using Jobtracker url.
> I am seeing for most of my reducing jobs, something like this :
>
> task_201201250352_0001_r_000000 31.05% reduce > copy (395 of 424 at 0.00
> MB/s) > 25-Jan-2012 03:54:06
>
> task_201201250352_0001_r_000001 30.73% reduce > copy (391 of 424 at 0.00
> MB/s) > 25-Jan-2012 03:54:06
>
> task_201201250352_0001_r_000002 30.89% reduce > copy (393 of 424 at 0.00
> MB/s) > 25-Jan-2012 03:54:06
>
>
> ............................
>
> Can anyone explain me why the speed is 0.00 MB/s. Job is running fine.
> Is it because this reduce > copy is happening on the same machine.
>
> Thanks,
> Praveenesh
--
Harsh J
Customer Ops. Engineer, Cloudera
Re: Reduce > copy at 0.00 MB/s
Posted by hadoop hive <ha...@gmail.com>.
this problem arise after adding a node , so then i start balancer to make
it balance ,
On Wed, Jan 25, 2012 at 4:38 PM, praveenesh kumar <pr...@gmail.com>wrote:
> @hadoophive
>
> Can you explain more by "balance the cluster" ?
>
> Thanks,
> Praveenesh
>
> On Wed, Jan 25, 2012 at 4:29 PM, hadoop hive <ha...@gmail.com> wrote:
>
> > i face the same issue but after sumtime when i balanced the cluster the
> > jobs started running fine,
> >
> > On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar <praveenesh@gmail.com
> > >wrote:
> >
> > > Hey
> > >
> > > Can anyone explain me what is reduce > copy phase in the reducer
> section
> > ?
> > > The (K,List(V)), is passed to the reducer. Is reduce > copy
> representing
> > > copying of K,List(V) on the reducer from all mappers ?
> > >
> > > I am monitoring my jobs on the cluster, using Jobtracker url.
> > > I am seeing for most of my reducing jobs, something like this :
> > >
> > > task_201201250352_0001_r_000000 31.05% reduce > copy (395 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > > task_201201250352_0001_r_000001 30.73% reduce > copy (391 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > > task_201201250352_0001_r_000002 30.89% reduce > copy (393 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > >
> > > ............................
> > >
> > > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine.
> > > Is it because this reduce > copy is happening on the same machine.
> > >
> > > Thanks,
> > > Praveenesh
> > >
> >
>
Re: Reduce > copy at 0.00 MB/s
Posted by praveenesh kumar <pr...@gmail.com>.
@hadoophive
Can you explain more by "balance the cluster" ?
Thanks,
Praveenesh
On Wed, Jan 25, 2012 at 4:29 PM, hadoop hive <ha...@gmail.com> wrote:
> i face the same issue but after sumtime when i balanced the cluster the
> jobs started running fine,
>
> On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar <praveenesh@gmail.com
> >wrote:
>
> > Hey
> >
> > Can anyone explain me what is reduce > copy phase in the reducer section
> ?
> > The (K,List(V)), is passed to the reducer. Is reduce > copy representing
> > copying of K,List(V) on the reducer from all mappers ?
> >
> > I am monitoring my jobs on the cluster, using Jobtracker url.
> > I am seeing for most of my reducing jobs, something like this :
> >
> > task_201201250352_0001_r_000000 31.05% reduce > copy (395 of 424 at
> 0.00
> > MB/s) > 25-Jan-2012 03:54:06
> >
> > task_201201250352_0001_r_000001 30.73% reduce > copy (391 of 424 at
> 0.00
> > MB/s) > 25-Jan-2012 03:54:06
> >
> > task_201201250352_0001_r_000002 30.89% reduce > copy (393 of 424 at
> 0.00
> > MB/s) > 25-Jan-2012 03:54:06
> >
> >
> > ............................
> >
> > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine.
> > Is it because this reduce > copy is happening on the same machine.
> >
> > Thanks,
> > Praveenesh
> >
>
Re: Reduce > copy at 0.00 MB/s
Posted by hadoop hive <ha...@gmail.com>.
i face the same issue but after sumtime when i balanced the cluster the
jobs started running fine,
On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar <pr...@gmail.com>wrote:
> Hey,
>
> Can anyone explain me what is reduce > copy phase in the reducer section ?
> The (K,List(V)), is passed to the reducer. Is reduce > copy representing
> copying of K,List(V) on the reducer from all mappers ?
>
> I am monitoring my jobs on the cluster, using Jobtracker url.
> I am seeing for most of my reducing jobs, something like this :
>
> task_201201250352_0001_r_000000 31.05% reduce > copy (395 of 424 at 0.00
> MB/s) > 25-Jan-2012 03:54:06
>
> task_201201250352_0001_r_000001 30.73% reduce > copy (391 of 424 at 0.00
> MB/s) > 25-Jan-2012 03:54:06
>
> task_201201250352_0001_r_000002 30.89% reduce > copy (393 of 424 at 0.00
> MB/s) > 25-Jan-2012 03:54:06
>
>
> ............................
>
> Can anyone explain me why the speed is 0.00 MB/s. Job is running fine.
> Is it because this reduce > copy is happening on the same machine.
>
> Thanks,
> Praveenesh
>