You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Milinda Pathirage <mp...@umail.iu.edu> on 2013/08/19 17:44:24 UTC
Re: Running Giraph on YARN
Hi Devs,
I was able to get Giraph running on Yarn by creating Yarn specific
configuration programmatically. I think that it's better if we can
have some shell scripts specific to Yarn (or may be modifications to
existing shell scripts), so that we can easily deploy Giraph jobs on
Yarn clusters. Please let me know if anyone is working on this. If no
one is working on that, I would like to work on that.
Thanks
Milinda
On Tue, Aug 13, 2013 at 2:33 PM, Milinda Pathirage
<mp...@umail.iu.edu> wrote:
> Hi,
>
> I'm trying to get Giraph running on YARN based on TestYarnJob test
> case. But having issues with moving required jars to YARN environment.
> I'm using single node YARN setup. I can see the job in YARN, but with
> following error.
>
> "Error: Could not find or load main class
> org.apache.giraph.yarn.GiraphApplicationMaster"
>
> I found that FileSystem.get(giraphConf) returns LocalFS inside
> resource copy methods by debugging. Can someone please point me to a
> doc or some writeup which describes how to properly configure
> GiraphYarnClient?
>
> Thanks
> Milinda
>
> --
> Milinda Pathirage
>
> twitter: milindalakmal
> skype: milinda.pathirage
> blog: http://milinda.pathirage.org
--
Milinda Pathirage
twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org
Re: Running Giraph on YARN
Posted by Eli Reisman <ap...@gmail.com>.
Sorry have not been checking my Apache mail enough. No I could do that,
I've needed to for a while. thanks!
Hope all is going well,
Eli
On Fri, Aug 23, 2013 at 1:49 AM, Claudio Martella <
claudio.martella@gmail.com> wrote:
> Also, a little documentation page could help newcomers to run jobs on YARN,
> if they have specific stuff.
> Eli, would that be a lot of work?
>
>
> On Fri, Aug 23, 2013 at 4:02 AM, Eli Reisman <apache.mailbox@gmail.com
> >wrote:
>
> > There are several YARN-specific configurations that must be applied when
> > launching YARN jobs. There are command line options for these, take a
> look
> > here:
> > giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java
> >
> > I think a Giraph on YARN launcher script is a great idea if the configs
> > make the command-line invocation too verbose.
> >
> > That being said, there is already a Giraph launcher script that could be
> > adapted to launch Giraph on YARN jobs (it launches only Non-YARN right
> > now.) Please feel free to put up a JIRA and post some patches!
> >
> >
> >
> > On Thu, Aug 22, 2013 at 3:58 PM, Armando Miraglia
> > <a....@student.vu.nl>wrote:
> >
> > > Hi Milinda,
> > >
> > > On Mon, Aug 19, 2013 at 11:44:24AM -0400, Milinda Pathirage wrote:
> > > > I was able to get Giraph running on Yarn by creating Yarn specific
> > > > configuration programmatically.
> > >
> > > If you could provide your solution that'd be great because I am having
> > > the same problem :)
> > >
> > > Thanks,
> > > Armando
> > >
> >
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
>
Re: Running Giraph on YARN
Posted by Claudio Martella <cl...@gmail.com>.
Also, a little documentation page could help newcomers to run jobs on YARN,
if they have specific stuff.
Eli, would that be a lot of work?
On Fri, Aug 23, 2013 at 4:02 AM, Eli Reisman <ap...@gmail.com>wrote:
> There are several YARN-specific configurations that must be applied when
> launching YARN jobs. There are command line options for these, take a look
> here:
> giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java
>
> I think a Giraph on YARN launcher script is a great idea if the configs
> make the command-line invocation too verbose.
>
> That being said, there is already a Giraph launcher script that could be
> adapted to launch Giraph on YARN jobs (it launches only Non-YARN right
> now.) Please feel free to put up a JIRA and post some patches!
>
>
>
> On Thu, Aug 22, 2013 at 3:58 PM, Armando Miraglia
> <a....@student.vu.nl>wrote:
>
> > Hi Milinda,
> >
> > On Mon, Aug 19, 2013 at 11:44:24AM -0400, Milinda Pathirage wrote:
> > > I was able to get Giraph running on Yarn by creating Yarn specific
> > > configuration programmatically.
> >
> > If you could provide your solution that'd be great because I am having
> > the same problem :)
> >
> > Thanks,
> > Armando
> >
>
--
Claudio Martella
claudio.martella@gmail.com
Re: Running Giraph on YARN
Posted by Eli Reisman <ap...@gmail.com>.
There are several YARN-specific configurations that must be applied when
launching YARN jobs. There are command line options for these, take a look
here: giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java
I think a Giraph on YARN launcher script is a great idea if the configs
make the command-line invocation too verbose.
That being said, there is already a Giraph launcher script that could be
adapted to launch Giraph on YARN jobs (it launches only Non-YARN right
now.) Please feel free to put up a JIRA and post some patches!
On Thu, Aug 22, 2013 at 3:58 PM, Armando Miraglia
<a....@student.vu.nl>wrote:
> Hi Milinda,
>
> On Mon, Aug 19, 2013 at 11:44:24AM -0400, Milinda Pathirage wrote:
> > I was able to get Giraph running on Yarn by creating Yarn specific
> > configuration programmatically.
>
> If you could provide your solution that'd be great because I am having
> the same problem :)
>
> Thanks,
> Armando
>
Re: Running Giraph on YARN
Posted by Armando Miraglia <a....@student.vu.nl>.
Hi Milinda,
On Mon, Aug 19, 2013 at 11:44:24AM -0400, Milinda Pathirage wrote:
> I was able to get Giraph running on Yarn by creating Yarn specific
> configuration programmatically.
If you could provide your solution that'd be great because I am having
the same problem :)
Thanks,
Armando