You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by mdk-swandha <di...@gmail.com> on 2017/03/06 19:49:17 UTC
Job Submission to Cross Cluster Fails
Hi,
I have the following configuration:
Cluster 1:
Job Submission through Hue (hue1) to Cluster 1 (NN1 and RM1) via Oozie
works.
Cluster 2:
Job Submission through Hue (hue2) to Cluster 2 (NN2 and RM2) via Oozie
works.
As Oozie is cluster agnostic - now I have tweaked my Hue code to submit job
to cross cluster i.e. from Hue1 to Cluster2 (NN2 and RM2). In the workspace
folder I do see job.properties with modified NN and JT (RM)
Job is going to RM2 but it is failing there with the following warnings and
error
Warning:
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override
final parameter: fs.defaultFS; Ignoring.
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override
final parameter: fs.defaultFS; Ignoring.
java.io.FileNotFoundException: File does not exist:
hdfs://cluster1-nn1:8020/user/user1/.staging/job_1488573599716_0001/job.splitmetainfo
org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1488573599716_0001_000001 not found in
AMRMTokenSecretManager.
Also it is trying to connect to RM1 as it shows the following message in the log
INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to
ResourceManager at RM1/xx.xx.xx.xx:8030
I have modified following in Oozie-site.xml
oozie.service.ProxyUserService.proxyuser.hue.groups ==> *
oozie.service.ProxyUserService.proxyuser.hue.hosts ==> *
I would appreciate if someone can shed some light on how to make this work.
Do I need to route job to cluster specific Oozie server?
Why job.xml (sent to RM via Oozie) has Cluster-1's NN and RM
information (if that is the case)?
Do I require any other configuration setting to enable this cross
cluster job submission work?
Thanks.
Re: Job Submission to Cross Cluster Fails
Posted by mdk-swandha <di...@gmail.com>.
Thanks for replying Peter.
I tried to submit the same job from command line on the oozie server i.e.
oozie job -oozie http://localhost:11000/oozie -config
examples/apps/spark/job.properties -run
I have modified job.properties and provided NN and RM of the second cluster.
I again got the same error -
1. WARN [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job
init failed
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.io.FileNotFoundException: File does not exist:
hdfs://NN1:8020/user/dkhakhkhar/.staging/job_1488573599716_0013/job.splitmetainfo
2. WARN [main] org.apache.hadoop.ipc.Client: Exception encountered while
connecting to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1488573599716_0013_000002 not found in
AMRMTokenSecretManager.
This is not a secured (kerberized) cluster.
I found that for each workflow submission Oozie creates this directory and file
/user/{username}/oozie-oozi/0000022-170222235043049-oozie-oozi-W/spark-node--spark/action.xml
In this file, I see it gets old NN1 (NN for the first cluster not
mentioned in the job.properties file) information programmatically for
the following properties
<property><name>mapreduce.job.classpath.files</name>
<property><name>dfs.namenode.rpc-address</name>
<property><name>dfs.namenode.https-address</name>
<property><name>mapreduce.job.cache.files</name>
<property><name>yarn.resourcemanager.zk-address</name>
<property><name>dfs.namenode.http-address</name>
<property><name>hadoop.registry.zk.quorum</name>
It also gets RM information programmatically which is for the first
cluster. Following properties I see in this file
<property><name>yarn.resourcemanager.resource-tracker.address</name>
<property><name>yarn.resourcemanager.admin.address</name><value>
<property><name>yarn.resourcemanager.webapp.address</name>
<property><name>yarn.resourcemanager.webapp.https.address</name>
<property><name>yarn.resourcemanager.hostname</name>
<property><name>yarn.resourcemanager.scheduler.address</name>
<property><name>yarn.resourcemanager.resource-tracker.address</name>
<property><name>yarn.timeline-service.address</name>
<property><name>yarn.timeline-service.webapp.https.address</name>
<property><name>yarn.timeline-service.webapp.address</name>
<property><name>mapreduce.jobhistory.webapp.address</name>
<property><name>mapreduce.jobhistory.address</name>
Does it get this information from the past run? Is it getting from job
history stored in Oozie db? Does it cache each past job and gets most
of the information from there? How can I override these settings i.e.
in other words when submitting job can Oozie do everything from
scratch? I thought by changing NN and RM in job.properties, an oozie
workflow will be able to run on any cluster but it doesn't seem to be
the case.
Thanks.
On Wed, Mar 8, 2017 at 1:51 PM, Peter Cseh <ge...@cloudera.com> wrote:
> Hi,
>
> Have you tried running dryruns to see the exact configurations the
> workflows and the actions are getting from Hue?
> Oozie is designed to work against multiple clusters, but I don't know if it
> can be done via Hue.
>
> gp
>
> On Mon, Mar 6, 2017 at 8:49 PM, mdk-swandha <di...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have the following configuration:
> >
> > Cluster 1:
> > Job Submission through Hue (hue1) to Cluster 1 (NN1 and RM1) via Oozie
> > works.
> >
> > Cluster 2:
> > Job Submission through Hue (hue2) to Cluster 2 (NN2 and RM2) via Oozie
> > works.
> >
> > As Oozie is cluster agnostic - now I have tweaked my Hue code to submit
> job
> > to cross cluster i.e. from Hue1 to Cluster2 (NN2 and RM2). In the
> workspace
> > folder I do see job.properties with modified NN and JT (RM)
> >
> > Job is going to RM2 but it is failing there with the following warnings
> and
> > error
> >
> > Warning:
> >
> > org.apache.hadoop.conf.Configuration: job.xml:an attempt to override
> > final parameter: fs.defaultFS; Ignoring.
> > org.apache.hadoop.conf.Configuration: job.xml:an attempt to override
> > final parameter: fs.defaultFS; Ignoring.
> >
> >
> > java.io.FileNotFoundException: File does not exist:
> > hdfs://cluster1-nn1:8020/user/user1/.staging/job_1488573599716_0001/job.
> > splitmetainfo
> >
> >
> > org.apache.hadoop.security.token.SecretManager$InvalidToken):
> > appattempt_1488573599716_0001_000001 not found in
> > AMRMTokenSecretManager.
> >
> >
> > Also it is trying to connect to RM1 as it shows the following message in
> > the log
> >
> >
> > INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to
> > ResourceManager at RM1/xx.xx.xx.xx:8030
> >
> >
> > I have modified following in Oozie-site.xml
> >
> >
> > oozie.service.ProxyUserService.proxyuser.hue.groups ==> *
> >
> > oozie.service.ProxyUserService.proxyuser.hue.hosts ==> *
> >
> >
> > I would appreciate if someone can shed some light on how to make this
> work.
> >
> >
> > Do I need to route job to cluster specific Oozie server?
> >
> >
> > Why job.xml (sent to RM via Oozie) has Cluster-1's NN and RM
> > information (if that is the case)?
> >
> >
> > Do I require any other configuration setting to enable this cross
> > cluster job submission work?
> >
> >
> > Thanks.
> >
>
>
>
> --
> Peter Cseh
> Software Engineer
> <http://www.cloudera.com>
>
Re: Job Submission to Cross Cluster Fails
Posted by Peter Cseh <ge...@cloudera.com>.
Hi,
Have you tried running dryruns to see the exact configurations the
workflows and the actions are getting from Hue?
Oozie is designed to work against multiple clusters, but I don't know if it
can be done via Hue.
gp
On Mon, Mar 6, 2017 at 8:49 PM, mdk-swandha <di...@gmail.com>
wrote:
> Hi,
>
> I have the following configuration:
>
> Cluster 1:
> Job Submission through Hue (hue1) to Cluster 1 (NN1 and RM1) via Oozie
> works.
>
> Cluster 2:
> Job Submission through Hue (hue2) to Cluster 2 (NN2 and RM2) via Oozie
> works.
>
> As Oozie is cluster agnostic - now I have tweaked my Hue code to submit job
> to cross cluster i.e. from Hue1 to Cluster2 (NN2 and RM2). In the workspace
> folder I do see job.properties with modified NN and JT (RM)
>
> Job is going to RM2 but it is failing there with the following warnings and
> error
>
> Warning:
>
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to override
> final parameter: fs.defaultFS; Ignoring.
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to override
> final parameter: fs.defaultFS; Ignoring.
>
>
> java.io.FileNotFoundException: File does not exist:
> hdfs://cluster1-nn1:8020/user/user1/.staging/job_1488573599716_0001/job.
> splitmetainfo
>
>
> org.apache.hadoop.security.token.SecretManager$InvalidToken):
> appattempt_1488573599716_0001_000001 not found in
> AMRMTokenSecretManager.
>
>
> Also it is trying to connect to RM1 as it shows the following message in
> the log
>
>
> INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to
> ResourceManager at RM1/xx.xx.xx.xx:8030
>
>
> I have modified following in Oozie-site.xml
>
>
> oozie.service.ProxyUserService.proxyuser.hue.groups ==> *
>
> oozie.service.ProxyUserService.proxyuser.hue.hosts ==> *
>
>
> I would appreciate if someone can shed some light on how to make this work.
>
>
> Do I need to route job to cluster specific Oozie server?
>
>
> Why job.xml (sent to RM via Oozie) has Cluster-1's NN and RM
> information (if that is the case)?
>
>
> Do I require any other configuration setting to enable this cross
> cluster job submission work?
>
>
> Thanks.
>
--
Peter Cseh
Software Engineer
<http://www.cloudera.com>