You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Jack Huang <ja...@machinezone.com> on 2016/02/25 02:56:33 UTC
Port issue with running hello-samza on HDP managed by Ambari
Hi all,
I am having trouble running the hello-samza example (
http://samza.apache.org/learn/tutorials/0.10/run-in-multi-node-yarn.html)
on our HDP cluster, which is set up with Ambari.
Upon running
*deploy/samza/bin/run-job.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties*
The output shows that Samza keeps trying to reach YARN's
ResourceManager at *0.0.0.0:8032
<http://0.0.0.0:8032>* but can't.
*2016-02-25 01:28:00 ClientHelper [INFO] trying to connect to RM
0.0.0.0:8032 <http://0.0.0.0:8032>*
*2016-02-25 01:28:00 RMProxy [INFO] Connecting to ResourceManager at
/0.0.0.0:8032 <http://0.0.0.0:8032>*
*2016-02-25 01:28:00 NativeCodeLoader [WARN] Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable*
*2016-02-25 01:28:02 Client [INFO] Retrying connect to server:
0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)*
*2016-02-25 01:28:03 Client [INFO] Retrying connect to server:
0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)*
*2016-02-25 01:28:04 Client [INFO] Retrying connect to server:
0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)*
For our cluster ResourceManager is running on *node2*. Here are the
properties related to the ResourceManager address in *yarn-site.xml *(which
was generated by Ambari) on the machine I am running hello-samza:
*<property>*
* <name>yarn.resourcemanager.address</name>*
* <value>node02:8050</value>*
*</property>*
*<property>*
* <name>yarn.resourcemanager.admin.address</name>*
* <value>node02:8141</value>*
*</property>*
*<property>*
* <name>yarn.resourcemanager.bind-host</name>*
* <value>0.0.0.0</value>*
*</property>*
*<property>*
* <name>yarn.resourcemanager.resource-tracker.address</name>*
* <value>node02:8025</value>*
*</property>*
*<property>*
* <name>yarn.resourcemanager.scheduler.address</name>*
* <value>node02:8030</value>*
*</property>*
*<property>*
* <name>yarn.resourcemanager.webapp.https.address</name>*
* <value>node02:8090</value>*
*</property>*
*<property>*
* <name>yarn.resourcemanager.webapp.address</name>*
* <value>node02:8088</value>*
*</property>*
*<property>*
* <name>yarn.resourcemanager.hostname</name>*
* <value>node02</value>*
*</property>*
*<property>*
* <name>yarn.resourcemanager.zk-address</name>*
* <value>node03:2181,node01:2181,node02:2181</value>*
*</property>*
I can't find any other place that specifies ResourceManager address. Can
anyone help?
Thanks,
Jack Huang
Re: Port issue with running hello-samza on HDP managed by Ambari
Posted by "Abdollahian Noghabi, Shadi" <ab...@illinois.edu>.
Hi,
I had the same issue. Apparently anything running on Hadoop 2.5 and later requires the jackson to be 1.9.13 an above. SAMZA-878 is about this issue. To fix it you have upgrade your jackson dependency in samza and your job to 1.9.13.
Shadi
-- Shadi Abdollahian
On Feb 26, 2016, at 10:28 PM, Նարեկ Գալստեան <ng...@gmail.com>> wrote:
Hi all,
I am trying to do the same thing but am having a different issue.
Job fails to successfully register with YARN which is caused by jackson library.
It seems that AMbari managed YARN for some reason requires Jackson 1.9.2 and above while Samza job needs Jackson 1.8.5 as JsonClass class used by Samza Json SerDe is missing from the next versions.
Has anyone encountered this problem, or can anyone help me resolve it?
Narek Galstyan
Նարեկ Գալստյան
On 27 February 2016 at 06:47, Jagadish Venkatraman <ja...@gmail.com>> wrote:
Hi jack,
Glad that you figured this out.
On Friday, February 26, 2016, Jack Huang <ja...@machinezone.com>> wrote:
> Turns out that I need to set the environment variable
>
> export HADOOP_YARN_HOME=/etc/hadoop
>
> so that the *run-class.sh *script will pick up the right config file
>
> Jack Huang
>
> On Wed, Feb 24, 2016 at 5:56 PM, Jack Huang <ja...@machinezone.com>
> <javascript:;>>
> wrote:
>
> > Hi all,
> >
> > I am having trouble running the hello-samza example (
> > http://samza.apache.org/learn/tutorials/0.10/run-in-multi-node-yarn.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__samza.apache.org_learn_tutorials_0.10_run-2Din-2Dmulti-2Dnode-2Dyarn.html&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=qG34-qVxYPjj0Nil1AMFJtdZMvH_ZnPDxWvhfgrs2m0&e=>
> )
> > on our HDP cluster, which is set up with Ambari.
> >
> > Upon running
> >
> > *deploy/samza/bin/run-job.sh
> >
> --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
> > --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties*
> >
> >
> > The output shows that Samza keeps trying to reach YARN's ResourceManager
> > at *0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=9qkihX0Z4RbBewb4SjTN2zWzjLqNO-dBc_J9y1TRMpg&e=> <http://0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=9qkihX0Z4RbBewb4SjTN2zWzjLqNO-dBc_J9y1TRMpg&e=>>* but can't.
> >
> > *2016-02-25 01:28:00 ClientHelper [INFO] trying to connect to RM
> > 0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=9qkihX0Z4RbBewb4SjTN2zWzjLqNO-dBc_J9y1TRMpg&e=> <http://0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=9qkihX0Z4RbBewb4SjTN2zWzjLqNO-dBc_J9y1TRMpg&e=>>*
> > *2016-02-25 01:28:00 RMProxy [INFO] Connecting to ResourceManager at
> > /0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=9qkihX0Z4RbBewb4SjTN2zWzjLqNO-dBc_J9y1TRMpg&e=> <http://0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=9qkihX0Z4RbBewb4SjTN2zWzjLqNO-dBc_J9y1TRMpg&e=>>*
> > *2016-02-25 01:28:00 NativeCodeLoader [WARN] Unable to load native-hadoop
> > library for your platform... using builtin-java classes where applicable*
> > *2016-02-25 01:28:02 Client [INFO] Retrying connect to server:
> > 0.0.0.0/0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=1cHJWLLqNfnD8jsr4xCv6lG47onecq3vC9wo6YZTnCk&e=> <http://0.0.0.0/0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=1cHJWLLqNfnD8jsr4xCv6lG47onecq3vC9wo6YZTnCk&e=>>. Already tried 0
> > time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> > sleepTime=1000 MILLISECONDS)*
> > *2016-02-25 01:28:03 Client [INFO] Retrying connect to server:
> > 0.0.0.0/0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=1cHJWLLqNfnD8jsr4xCv6lG47onecq3vC9wo6YZTnCk&e=> <http://0.0.0.0/0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=1cHJWLLqNfnD8jsr4xCv6lG47onecq3vC9wo6YZTnCk&e=>>. Already tried 1
> > time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> > sleepTime=1000 MILLISECONDS)*
> > *2016-02-25 01:28:04 Client [INFO] Retrying connect to server:
> > 0.0.0.0/0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=1cHJWLLqNfnD8jsr4xCv6lG47onecq3vC9wo6YZTnCk&e=> <http://0.0.0.0/0.0.0.0:8032<https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A8032&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=qBMNnXfZNvyss4rX0zSLrT-SXVl3iZAwnU7VGr4eeeE&m=Q06Q8gR1nsq7lLp2AZ2POaq2VvvSyDrsaGwCpx6xnyE&s=1cHJWLLqNfnD8jsr4xCv6lG47onecq3vC9wo6YZTnCk&e=>>. Already tried 2
> > time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> > sleepTime=1000 MILLISECONDS)*
> >
> >
> > For our cluster ResourceManager is running on *node2*. Here are the
> > properties related to the ResourceManager address in *yarn-site.xml
> *(which
> > was generated by Ambari) on the machine I am running hello-samza:
> >
> > *<property>*
> > * <name>yarn.resourcemanager.address</name>*
> > * <value>node02:8050</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.admin.address</name>*
> > * <value>node02:8141</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.bind-host</name>*
> > * <value>0.0.0.0</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.resource-tracker.address</name>*
> > * <value>node02:8025</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.scheduler.address</name>*
> > * <value>node02:8030</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.webapp.https.address</name>*
> > * <value>node02:8090</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.webapp.address</name>*
> > * <value>node02:8088</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.hostname</name>*
> > * <value>node02</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.zk-address</name>*
> > * <value>node03:2181,node01:2181,node02:2181</value>*
> > *</property>*
> >
> >
> > I can't find any other place that specifies ResourceManager address. Can
> > anyone help?
> >
> >
> > Thanks,
> >
> > Jack Huang
> >
>
--
Sent from my iphone.
Re: Port issue with running hello-samza on HDP managed by Ambari
Posted by Նարեկ Գալստեան <ng...@gmail.com>.
Hi all,
I am trying to do the same thing but am having a different issue.
Job fails to successfully register with YARN which is caused by jackson
library.
It seems that AMbari managed YARN for some reason requires Jackson 1.9.2
and above while Samza job needs Jackson 1.8.5 as JsonClass class used by
Samza Json SerDe is missing from the next versions.
Has anyone encountered this problem, or can anyone help me resolve it?
Narek Galstyan
Նարեկ Գալստյան
On 27 February 2016 at 06:47, Jagadish Venkatraman <ja...@gmail.com>
wrote:
> Hi jack,
>
> Glad that you figured this out.
>
> On Friday, February 26, 2016, Jack Huang <ja...@machinezone.com>
> wrote:
>
> > Turns out that I need to set the environment variable
> >
> > export HADOOP_YARN_HOME=/etc/hadoop
> >
> > so that the *run-class.sh *script will pick up the right config file
> >
> > Jack Huang
> >
> > On Wed, Feb 24, 2016 at 5:56 PM, Jack Huang <jackhuang@machinezone.com
> > <javascript:;>>
> > wrote:
> >
> > > Hi all,
> > >
> > > I am having trouble running the hello-samza example (
> > >
> http://samza.apache.org/learn/tutorials/0.10/run-in-multi-node-yarn.html
> > )
> > > on our HDP cluster, which is set up with Ambari.
> > >
> > > Upon running
> > >
> > > *deploy/samza/bin/run-job.sh
> > >
> >
> --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
> > >
> --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties*
> > >
> > >
> > > The output shows that Samza keeps trying to reach YARN's
> ResourceManager
> > > at *0.0.0.0:8032 <http://0.0.0.0:8032>* but can't.
> > >
> > > *2016-02-25 01:28:00 ClientHelper [INFO] trying to connect to RM
> > > 0.0.0.0:8032 <http://0.0.0.0:8032>*
> > > *2016-02-25 01:28:00 RMProxy [INFO] Connecting to ResourceManager at
> > > /0.0.0.0:8032 <http://0.0.0.0:8032>*
> > > *2016-02-25 01:28:00 NativeCodeLoader [WARN] Unable to load
> native-hadoop
> > > library for your platform... using builtin-java classes where
> applicable*
> > > *2016-02-25 01:28:02 Client [INFO] Retrying connect to server:
> > > 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 0
> > > time(s); retry policy is
> > RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> > > sleepTime=1000 MILLISECONDS)*
> > > *2016-02-25 01:28:03 Client [INFO] Retrying connect to server:
> > > 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 1
> > > time(s); retry policy is
> > RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> > > sleepTime=1000 MILLISECONDS)*
> > > *2016-02-25 01:28:04 Client [INFO] Retrying connect to server:
> > > 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 2
> > > time(s); retry policy is
> > RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> > > sleepTime=1000 MILLISECONDS)*
> > >
> > >
> > > For our cluster ResourceManager is running on *node2*. Here are the
> > > properties related to the ResourceManager address in *yarn-site.xml
> > *(which
> > > was generated by Ambari) on the machine I am running hello-samza:
> > >
> > > *<property>*
> > > * <name>yarn.resourcemanager.address</name>*
> > > * <value>node02:8050</value>*
> > > *</property>*
> > > *<property>*
> > > * <name>yarn.resourcemanager.admin.address</name>*
> > > * <value>node02:8141</value>*
> > > *</property>*
> > > *<property>*
> > > * <name>yarn.resourcemanager.bind-host</name>*
> > > * <value>0.0.0.0</value>*
> > > *</property>*
> > > *<property>*
> > > * <name>yarn.resourcemanager.resource-tracker.address</name>*
> > > * <value>node02:8025</value>*
> > > *</property>*
> > > *<property>*
> > > * <name>yarn.resourcemanager.scheduler.address</name>*
> > > * <value>node02:8030</value>*
> > > *</property>*
> > > *<property>*
> > > * <name>yarn.resourcemanager.webapp.https.address</name>*
> > > * <value>node02:8090</value>*
> > > *</property>*
> > > *<property>*
> > > * <name>yarn.resourcemanager.webapp.address</name>*
> > > * <value>node02:8088</value>*
> > > *</property>*
> > > *<property>*
> > > * <name>yarn.resourcemanager.hostname</name>*
> > > * <value>node02</value>*
> > > *</property>*
> > > *<property>*
> > > * <name>yarn.resourcemanager.zk-address</name>*
> > > * <value>node03:2181,node01:2181,node02:2181</value>*
> > > *</property>*
> > >
> > >
> > > I can't find any other place that specifies ResourceManager address.
> Can
> > > anyone help?
> > >
> > >
> > > Thanks,
> > >
> > > Jack Huang
> > >
> >
>
>
> --
> Sent from my iphone.
>
Re: Port issue with running hello-samza on HDP managed by Ambari
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Hi jack,
Glad that you figured this out.
On Friday, February 26, 2016, Jack Huang <ja...@machinezone.com> wrote:
> Turns out that I need to set the environment variable
>
> export HADOOP_YARN_HOME=/etc/hadoop
>
> so that the *run-class.sh *script will pick up the right config file
>
> Jack Huang
>
> On Wed, Feb 24, 2016 at 5:56 PM, Jack Huang <jackhuang@machinezone.com
> <javascript:;>>
> wrote:
>
> > Hi all,
> >
> > I am having trouble running the hello-samza example (
> > http://samza.apache.org/learn/tutorials/0.10/run-in-multi-node-yarn.html
> )
> > on our HDP cluster, which is set up with Ambari.
> >
> > Upon running
> >
> > *deploy/samza/bin/run-job.sh
> >
> --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
> > --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties*
> >
> >
> > The output shows that Samza keeps trying to reach YARN's ResourceManager
> > at *0.0.0.0:8032 <http://0.0.0.0:8032>* but can't.
> >
> > *2016-02-25 01:28:00 ClientHelper [INFO] trying to connect to RM
> > 0.0.0.0:8032 <http://0.0.0.0:8032>*
> > *2016-02-25 01:28:00 RMProxy [INFO] Connecting to ResourceManager at
> > /0.0.0.0:8032 <http://0.0.0.0:8032>*
> > *2016-02-25 01:28:00 NativeCodeLoader [WARN] Unable to load native-hadoop
> > library for your platform... using builtin-java classes where applicable*
> > *2016-02-25 01:28:02 Client [INFO] Retrying connect to server:
> > 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 0
> > time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> > sleepTime=1000 MILLISECONDS)*
> > *2016-02-25 01:28:03 Client [INFO] Retrying connect to server:
> > 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 1
> > time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> > sleepTime=1000 MILLISECONDS)*
> > *2016-02-25 01:28:04 Client [INFO] Retrying connect to server:
> > 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 2
> > time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> > sleepTime=1000 MILLISECONDS)*
> >
> >
> > For our cluster ResourceManager is running on *node2*. Here are the
> > properties related to the ResourceManager address in *yarn-site.xml
> *(which
> > was generated by Ambari) on the machine I am running hello-samza:
> >
> > *<property>*
> > * <name>yarn.resourcemanager.address</name>*
> > * <value>node02:8050</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.admin.address</name>*
> > * <value>node02:8141</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.bind-host</name>*
> > * <value>0.0.0.0</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.resource-tracker.address</name>*
> > * <value>node02:8025</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.scheduler.address</name>*
> > * <value>node02:8030</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.webapp.https.address</name>*
> > * <value>node02:8090</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.webapp.address</name>*
> > * <value>node02:8088</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.hostname</name>*
> > * <value>node02</value>*
> > *</property>*
> > *<property>*
> > * <name>yarn.resourcemanager.zk-address</name>*
> > * <value>node03:2181,node01:2181,node02:2181</value>*
> > *</property>*
> >
> >
> > I can't find any other place that specifies ResourceManager address. Can
> > anyone help?
> >
> >
> > Thanks,
> >
> > Jack Huang
> >
>
--
Sent from my iphone.
Re: Port issue with running hello-samza on HDP managed by Ambari
Posted by Jack Huang <ja...@machinezone.com>.
Turns out that I need to set the environment variable
export HADOOP_YARN_HOME=/etc/hadoop
so that the *run-class.sh *script will pick up the right config file
Jack Huang
On Wed, Feb 24, 2016 at 5:56 PM, Jack Huang <ja...@machinezone.com>
wrote:
> Hi all,
>
> I am having trouble running the hello-samza example (
> http://samza.apache.org/learn/tutorials/0.10/run-in-multi-node-yarn.html)
> on our HDP cluster, which is set up with Ambari.
>
> Upon running
>
> *deploy/samza/bin/run-job.sh
> --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
> --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties*
>
>
> The output shows that Samza keeps trying to reach YARN's ResourceManager
> at *0.0.0.0:8032 <http://0.0.0.0:8032>* but can't.
>
> *2016-02-25 01:28:00 ClientHelper [INFO] trying to connect to RM
> 0.0.0.0:8032 <http://0.0.0.0:8032>*
> *2016-02-25 01:28:00 RMProxy [INFO] Connecting to ResourceManager at
> /0.0.0.0:8032 <http://0.0.0.0:8032>*
> *2016-02-25 01:28:00 NativeCodeLoader [WARN] Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable*
> *2016-02-25 01:28:02 Client [INFO] Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 0
> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)*
> *2016-02-25 01:28:03 Client [INFO] Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 1
> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)*
> *2016-02-25 01:28:04 Client [INFO] Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 2
> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)*
>
>
> For our cluster ResourceManager is running on *node2*. Here are the
> properties related to the ResourceManager address in *yarn-site.xml *(which
> was generated by Ambari) on the machine I am running hello-samza:
>
> *<property>*
> * <name>yarn.resourcemanager.address</name>*
> * <value>node02:8050</value>*
> *</property>*
> *<property>*
> * <name>yarn.resourcemanager.admin.address</name>*
> * <value>node02:8141</value>*
> *</property>*
> *<property>*
> * <name>yarn.resourcemanager.bind-host</name>*
> * <value>0.0.0.0</value>*
> *</property>*
> *<property>*
> * <name>yarn.resourcemanager.resource-tracker.address</name>*
> * <value>node02:8025</value>*
> *</property>*
> *<property>*
> * <name>yarn.resourcemanager.scheduler.address</name>*
> * <value>node02:8030</value>*
> *</property>*
> *<property>*
> * <name>yarn.resourcemanager.webapp.https.address</name>*
> * <value>node02:8090</value>*
> *</property>*
> *<property>*
> * <name>yarn.resourcemanager.webapp.address</name>*
> * <value>node02:8088</value>*
> *</property>*
> *<property>*
> * <name>yarn.resourcemanager.hostname</name>*
> * <value>node02</value>*
> *</property>*
> *<property>*
> * <name>yarn.resourcemanager.zk-address</name>*
> * <value>node03:2181,node01:2181,node02:2181</value>*
> *</property>*
>
>
> I can't find any other place that specifies ResourceManager address. Can
> anyone help?
>
>
> Thanks,
>
> Jack Huang
>