You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Eli Reisman <ap...@gmail.com> on 2015/02/08 20:46:35 UTC
Re: What is the best recipe for determining maximum worker count?
That's correct. There were some changes last year as to. How these worker
containers are allocated that were supposed to fix another similar issue
with the master task. Perhaps this introduced another bug. I'm not sure
there's enough info to be sure can you give more details? Anyone else
seeing this issue? And to confirm this was only when you run on a single
machine?
On Jan 17, 2015 12:09 PM, "Claudio Martella" <cl...@gmail.com>
wrote:
> as far as I understand, with YARN there are two additional task to the
> workers: one for the master and one for the application manager. I'm not
> sure, but maybe Eli can confirm.
>
> On Fri, Jan 16, 2015 at 2:16 PM, Dongjin Lee <do...@gmail.com>
> wrote:
>
> > * I apologize in advance for my bad English.
> >
> > Hello. I have a question about determining optimal worker count. I have
> > been trying SimpleShortestPathsComputation example with Hadoop 2.5.1. And
> > noticed that the maximum value for -w parameter differs per
> configuration.
> >
> > Let me explain: I have been running Giraph jobs on my local Ubuntu
> machine,
> > with Intel i7 cpu (4core) and 16gb ram. I can run 8 threads
> simultaneously
> > and has 9gb of free memory.
> >
> > First, I configured Hadoop can run 4 mappers with 2gb of memory for each.
> > When I ran SimpleShortestPathsComputation, w=2 worked well but w=3 did
> not
> > work - It froze with 'map 75%'. That is, I could not use 2 mappers for
> > worker.
> >
> > After that, I re-configured my Hadoop distribution to run 8 mappers with
> > 1gb of memory for each. When I re-ran SimpleShortestPathsComputation, w=5
> > worked well but with w=6 it froze with 'map 86%'. In this case, I could
> not
> > use 3 mappers for worker.
> >
> > I already know that there is an additional mapper, which works as
> > BSPMaster. However, It is still mysterious for me: Is there any hidden
> > process I don't know? Then, how many are there? 1 or 2?
> >
> > Any guidance will be welcomed.
> >
> > Thanks in advance,
> > Dongjin
> >
> > PS - When I was working with Hadoop 1.x.x, I calculated the maximum
> worker
> > count as {available mappers} - 1 and it worked perfectly. I guess there
> are
> > some differences between 1.x.x and 2.x.x that I don't know.
> >
> > --
> > *Dongjin Lee*
> >
> >
> > *Oracle Server Technologies Group.So interested in massive-scale machine
> > learning.facebook: www.facebook.com/dongjin.lee.kr
> > <http://www.facebook.com/dongjin.lee.kr>linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > <http://kr.linkedin.com/in/dongjinleekr>github:
> > <http://goog_969573159>github.com/dongjinleekr
> > <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
> > <http://www.twitter.com/dongjinleekr>*
> >
>
>
>
> --
> Claudio Martella
>
Re: What is the best recipe for determining maximum worker count?
Posted by Dongjin Lee <do...@gmail.com>.
OK. I will take more experiments and let you know.
Thanks,
Dongjin
On Mon, Feb 9, 2015 at 4:46 AM, Eli Reisman <ap...@gmail.com>
wrote:
> That's correct. There were some changes last year as to. How these worker
> containers are allocated that were supposed to fix another similar issue
> with the master task. Perhaps this introduced another bug. I'm not sure
> there's enough info to be sure can you give more details? Anyone else
> seeing this issue? And to confirm this was only when you run on a single
> machine?
> On Jan 17, 2015 12:09 PM, "Claudio Martella" <cl...@gmail.com>
> wrote:
>
> > as far as I understand, with YARN there are two additional task to the
> > workers: one for the master and one for the application manager. I'm not
> > sure, but maybe Eli can confirm.
> >
> > On Fri, Jan 16, 2015 at 2:16 PM, Dongjin Lee <do...@gmail.com>
> > wrote:
> >
> > > * I apologize in advance for my bad English.
> > >
> > > Hello. I have a question about determining optimal worker count. I have
> > > been trying SimpleShortestPathsComputation example with Hadoop 2.5.1.
> And
> > > noticed that the maximum value for -w parameter differs per
> > configuration.
> > >
> > > Let me explain: I have been running Giraph jobs on my local Ubuntu
> > machine,
> > > with Intel i7 cpu (4core) and 16gb ram. I can run 8 threads
> > simultaneously
> > > and has 9gb of free memory.
> > >
> > > First, I configured Hadoop can run 4 mappers with 2gb of memory for
> each.
> > > When I ran SimpleShortestPathsComputation, w=2 worked well but w=3 did
> > not
> > > work - It froze with 'map 75%'. That is, I could not use 2 mappers for
> > > worker.
> > >
> > > After that, I re-configured my Hadoop distribution to run 8 mappers
> with
> > > 1gb of memory for each. When I re-ran SimpleShortestPathsComputation,
> w=5
> > > worked well but with w=6 it froze with 'map 86%'. In this case, I could
> > not
> > > use 3 mappers for worker.
> > >
> > > I already know that there is an additional mapper, which works as
> > > BSPMaster. However, It is still mysterious for me: Is there any hidden
> > > process I don't know? Then, how many are there? 1 or 2?
> > >
> > > Any guidance will be welcomed.
> > >
> > > Thanks in advance,
> > > Dongjin
> > >
> > > PS - When I was working with Hadoop 1.x.x, I calculated the maximum
> > worker
> > > count as {available mappers} - 1 and it worked perfectly. I guess there
> > are
> > > some differences between 1.x.x and 2.x.x that I don't know.
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > >
> > > *Oracle Server Technologies Group.So interested in massive-scale
> machine
> > > learning.facebook: www.facebook.com/dongjin.lee.kr
> > > <http://www.facebook.com/dongjin.lee.kr>linkedin:
> > > kr.linkedin.com/in/dongjinleekr
> > > <http://kr.linkedin.com/in/dongjinleekr>github:
> > > <http://goog_969573159>github.com/dongjinleekr
> > > <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
> > > <http://www.twitter.com/dongjinleekr>*
> > >
> >
> >
> >
> > --
> > Claudio Martella
> >
>
--
*Dongjin Lee*
*Oracle Server Technologies Group.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
<http://www.facebook.com/dongjin.lee.kr>linkedin:
kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>github:
<http://goog_969573159>github.com/dongjinleekr
<http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
<http://www.twitter.com/dongjinleekr>*