You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by George Li <gl...@ca.ibm.com> on 2015/06/22 23:11:45 UTC
Deploying samza job to a remote yarn resource manager
Hi,
According to
https://samza.apache.org/learn/documentation/0.9/jobs/yarn-jobs.html
, to deploy samza jobs to a remote yarn resource manager, I need to set
the HADOOPYARNHOME environment variable. What should be the value of this
variable if my yarn-site.xml is at
$HOME/hadoop-2.6.0/etc/hadoop/yarn-site.xml?
Thanks,
George
Re: Deploying samza job to a remote yarn resource manager
Posted by Navina Ramesh <nr...@linkedin.com.INVALID>.
Hi George,
JobRunner is meant to run on the localhost only. You have write a wrapper
script (perhaps in python or bash) to ssh to the RM and then run the
JobRunner. RM machine should already have HADOOP_YARN_HOME and
HADOOP_YARN_CONF_DIR configured, if the RM is running. So, probably not
necessary to include in the classpath.
Cheers,
Navina
On 6/23/15, 8:44 AM, "George Li" <gl...@ca.ibm.com> wrote:
>Hi Navina and Milinda,
>
>Thanks for your reply. I get it working by setting HADOOP_YARN_HOME and
>HADOOP_CONF_DIR
>
>export HADOOP_YARN_HOME=$HOME/hadoop-2.6.0/
>export HADOOP_CONF_DIR=$HOME/hadoop-2.6.0/etc/hadoop
>
>,i.e., similar to what run-class.sh does
>
>
>I am also trying to figure out how to deploy samza job by code. By
>porting
>over org.apache.samza.job.JobRunner, I am able to deploy a job as long as
>yarn resource manager is on the same host, but how do I pass the remote
>resource manager address? Maybe I should include HADOOP_YARN_HOME and
>HADOOP_CONF_DIR as classpath?
>
>George
>
>
>
>
>From: Navina Ramesh <nr...@linkedin.com.INVALID>
>To: "dev@samza.apache.org" <de...@samza.apache.org>,
>Date: 22/06/2015 05:57 PM
>Subject: Re: Deploying samza job to a remote yarn resource manager
>
>
>
>Hi George,
>
>Looks like there is a formatting issue on the samza website. The
>environment variable name should be HADOOP_YARN_HOME.
>In your case, it should point to hadoop home: $HOME/hadoop-2.6.0/
>
>Please let me know if it works.
>
>Cheers!
>Navina
>
>On 6/22/15, 2:11 PM, "George Li" <gl...@ca.ibm.com> wrote:
>
>>Hi,
>>
>>According to
>>
>>https://samza.apache.org/learn/documentation/0.9/jobs/yarn-jobs.html
>>
>>, to deploy samza jobs to a remote yarn resource manager, I need to set
>>the HADOOPYARNHOME environment variable. What should be the value of this
>>variable if my yarn-site.xml is at
>>$HOME/hadoop-2.6.0/etc/hadoop/yarn-site.xml?
>>
>>Thanks,
>>
>>George
>
>
Re: Deploying samza job to a remote yarn resource manager
Posted by George Li <gl...@ca.ibm.com>.
Hi Navina and Milinda,
Thanks for your reply. I get it working by setting HADOOP_YARN_HOME and
HADOOP_CONF_DIR
export HADOOP_YARN_HOME=$HOME/hadoop-2.6.0/
export HADOOP_CONF_DIR=$HOME/hadoop-2.6.0/etc/hadoop
,i.e., similar to what run-class.sh does
I am also trying to figure out how to deploy samza job by code. By porting
over org.apache.samza.job.JobRunner, I am able to deploy a job as long as
yarn resource manager is on the same host, but how do I pass the remote
resource manager address? Maybe I should include HADOOP_YARN_HOME and
HADOOP_CONF_DIR as classpath?
George
From: Navina Ramesh <nr...@linkedin.com.INVALID>
To: "dev@samza.apache.org" <de...@samza.apache.org>,
Date: 22/06/2015 05:57 PM
Subject: Re: Deploying samza job to a remote yarn resource manager
Hi George,
Looks like there is a formatting issue on the samza website. The
environment variable name should be HADOOP_YARN_HOME.
In your case, it should point to hadoop home: $HOME/hadoop-2.6.0/
Please let me know if it works.
Cheers!
Navina
On 6/22/15, 2:11 PM, "George Li" <gl...@ca.ibm.com> wrote:
>Hi,
>
>According to
>
>https://samza.apache.org/learn/documentation/0.9/jobs/yarn-jobs.html
>
>, to deploy samza jobs to a remote yarn resource manager, I need to set
>the HADOOPYARNHOME environment variable. What should be the value of this
>variable if my yarn-site.xml is at
>$HOME/hadoop-2.6.0/etc/hadoop/yarn-site.xml?
>
>Thanks,
>
>George
Re: Deploying samza job to a remote yarn resource manager
Posted by Milinda Pathirage <mp...@umail.iu.edu>.
Hi George,
Also have a look at
https://samza.apache.org/learn/tutorials/0.9/run-in-multi-node-yarn.html.
You may need to install HTTP file system as well. Let us know if you have
any questions.
Thanks
Milinda
On Mon, Jun 22, 2015 at 5:56 PM, Navina Ramesh <nramesh@linkedin.com.invalid
> wrote:
> Hi George,
>
> Looks like there is a formatting issue on the samza website. The
> environment variable name should be HADOOP_YARN_HOME.
> In your case, it should point to hadoop home: $HOME/hadoop-2.6.0/
>
> Please let me know if it works.
>
> Cheers!
> Navina
>
> On 6/22/15, 2:11 PM, "George Li" <gl...@ca.ibm.com> wrote:
>
> >Hi,
> >
> >According to
> >
> >https://samza.apache.org/learn/documentation/0.9/jobs/yarn-jobs.html
> >
> >, to deploy samza jobs to a remote yarn resource manager, I need to set
> >the HADOOPYARNHOME environment variable. What should be the value of this
> >variable if my yarn-site.xml is at
> >$HOME/hadoop-2.6.0/etc/hadoop/yarn-site.xml?
> >
> >Thanks,
> >
> >George
>
>
--
Milinda Pathirage
PhD Student | Research Assistant
School of Informatics and Computing | Data to Insight Center
Indiana University
twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org
Re: Deploying samza job to a remote yarn resource manager
Posted by Navina Ramesh <nr...@linkedin.com.INVALID>.
Hi George,
Looks like there is a formatting issue on the samza website. The
environment variable name should be HADOOP_YARN_HOME.
In your case, it should point to hadoop home: $HOME/hadoop-2.6.0/
Please let me know if it works.
Cheers!
Navina
On 6/22/15, 2:11 PM, "George Li" <gl...@ca.ibm.com> wrote:
>Hi,
>
>According to
>
>https://samza.apache.org/learn/documentation/0.9/jobs/yarn-jobs.html
>
>, to deploy samza jobs to a remote yarn resource manager, I need to set
>the HADOOPYARNHOME environment variable. What should be the value of this
>variable if my yarn-site.xml is at
>$HOME/hadoop-2.6.0/etc/hadoop/yarn-site.xml?
>
>Thanks,
>
>George