You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Siddhi Mehta <sm...@gmail.com> on 2013/09/18 21:37:39 UTC
Yarn Exception while getting JobStatus
Hey All,
I am running a Hadoop 2.0(cdh4.2.1) cluster.
>From the job client whenever we try to get the job status from the cluster
we run into the following IOException.
The exception has no message on the cause of the IOException.
Any ideas on what could be causing this issue.
Looking at the code it seems like it fails while talking to the
jobhistoryserver.
The application where the job client runs has most of the ports blocked.
We opened a the following ports for the job client to talk to the
cluster(8030,8031,8032
8033,19888, 10020) but still see the exception.
Is there a defined list of ports on which a mapreduce client communicates
with the Hadoop Cluster
Cause1: java.io.IOException
Cause1-StackTrace:
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(
ClientServiceDelegate.java:317)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(
ClientServiceDelegate.java:385)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(
YARNRunner.java:487)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:635)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:633)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(
UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(
JobClient.java:633)
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:651)
Thanks,
Siddhi
Re: Yarn Exception while getting JobStatus
Posted by Siddhi Mehta <sm...@gmail.com>.
Hey Harsh,
Here is the more complete stacktrace. I had truncated it earlier since it
was application specific.
Let me know if this helps.
Thrown: java.lang.RuntimeException java.io.IOException
Thrown-StackTrace:
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:37)
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:19)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Cause0: java.lang.RuntimeException java.io.IOException
Cause0-StackTrace:
at
hadoop.client.RemoteHadoopJobClientImpl.getJobStatus(RemoteHadoopJobClientImpl.java:45)
at HadoopJobUtil.updateRunningHadoopJobs(HadoopJobUtil.java:260)
at HadoopJobUpdaterProcess.executeInner(HadoopJobUpdaterProcess.java:54)
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:35)
... 6 shared with parent
Cause1: java.io.IOException
Cause1-StackTrace:
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:487)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:635)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:633)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:633)
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:651)
at
hadoop.client.RemoteHadoopJobClientImpl.getJobStatus(RemoteHadoopJobClientImpl.java:39)
... 16 shared with parent
On Wed, Sep 18, 2013 at 6:36 PM, Harsh J <ha...@cloudera.com> wrote:
> Please post your complete stacktrace - the provided one appears
> incomplete/truncated.
>
> On Thu, Sep 19, 2013 at 1:07 AM, Siddhi Mehta <sm...@gmail.com>
> wrote:
> > Hey All,
> >
> > I am running a Hadoop 2.0(cdh4.2.1) cluster.
> >
> > From the job client whenever we try to get the job status from the
> cluster
> > we run into the following IOException.
> > The exception has no message on the cause of the IOException.
> > Any ideas on what could be causing this issue.
> >
> > Looking at the code it seems like it fails while talking to the
> > jobhistoryserver.
> > The application where the job client runs has most of the ports blocked.
> > We opened a the following ports for the job client to talk to the
> > cluster(8030,8031,8032 8033,19888, 10020) but still see the exception.
> > Is there a defined list of ports on which a mapreduce client communicates
> > with the Hadoop Cluster
> >
> > Cause1: java.io.IOException
> > Cause1-StackTrace:
> > at
> >
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
> > at
> >
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385)
> > at
> > org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:487)
> > at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:635)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:633)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > at
> > org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:633)
> > at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:651)
> >
> > Thanks,
> > Siddhi
>
>
>
> --
> Harsh J
>
Re: Yarn Exception while getting JobStatus
Posted by Siddhi Mehta <sm...@gmail.com>.
Hey Harsh,
Here is the more complete stacktrace. I had truncated it earlier since it
was application specific.
Let me know if this helps.
Thrown: java.lang.RuntimeException java.io.IOException
Thrown-StackTrace:
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:37)
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:19)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Cause0: java.lang.RuntimeException java.io.IOException
Cause0-StackTrace:
at
hadoop.client.RemoteHadoopJobClientImpl.getJobStatus(RemoteHadoopJobClientImpl.java:45)
at HadoopJobUtil.updateRunningHadoopJobs(HadoopJobUtil.java:260)
at HadoopJobUpdaterProcess.executeInner(HadoopJobUpdaterProcess.java:54)
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:35)
... 6 shared with parent
Cause1: java.io.IOException
Cause1-StackTrace:
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:487)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:635)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:633)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:633)
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:651)
at
hadoop.client.RemoteHadoopJobClientImpl.getJobStatus(RemoteHadoopJobClientImpl.java:39)
... 16 shared with parent
On Wed, Sep 18, 2013 at 6:36 PM, Harsh J <ha...@cloudera.com> wrote:
> Please post your complete stacktrace - the provided one appears
> incomplete/truncated.
>
> On Thu, Sep 19, 2013 at 1:07 AM, Siddhi Mehta <sm...@gmail.com>
> wrote:
> > Hey All,
> >
> > I am running a Hadoop 2.0(cdh4.2.1) cluster.
> >
> > From the job client whenever we try to get the job status from the
> cluster
> > we run into the following IOException.
> > The exception has no message on the cause of the IOException.
> > Any ideas on what could be causing this issue.
> >
> > Looking at the code it seems like it fails while talking to the
> > jobhistoryserver.
> > The application where the job client runs has most of the ports blocked.
> > We opened a the following ports for the job client to talk to the
> > cluster(8030,8031,8032 8033,19888, 10020) but still see the exception.
> > Is there a defined list of ports on which a mapreduce client communicates
> > with the Hadoop Cluster
> >
> > Cause1: java.io.IOException
> > Cause1-StackTrace:
> > at
> >
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
> > at
> >
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385)
> > at
> > org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:487)
> > at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:635)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:633)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > at
> > org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:633)
> > at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:651)
> >
> > Thanks,
> > Siddhi
>
>
>
> --
> Harsh J
>
Re: Yarn Exception while getting JobStatus
Posted by Siddhi Mehta <sm...@gmail.com>.
Hey Harsh,
Here is the more complete stacktrace. I had truncated it earlier since it
was application specific.
Let me know if this helps.
Thrown: java.lang.RuntimeException java.io.IOException
Thrown-StackTrace:
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:37)
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:19)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Cause0: java.lang.RuntimeException java.io.IOException
Cause0-StackTrace:
at
hadoop.client.RemoteHadoopJobClientImpl.getJobStatus(RemoteHadoopJobClientImpl.java:45)
at HadoopJobUtil.updateRunningHadoopJobs(HadoopJobUtil.java:260)
at HadoopJobUpdaterProcess.executeInner(HadoopJobUpdaterProcess.java:54)
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:35)
... 6 shared with parent
Cause1: java.io.IOException
Cause1-StackTrace:
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:487)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:635)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:633)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:633)
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:651)
at
hadoop.client.RemoteHadoopJobClientImpl.getJobStatus(RemoteHadoopJobClientImpl.java:39)
... 16 shared with parent
On Wed, Sep 18, 2013 at 6:36 PM, Harsh J <ha...@cloudera.com> wrote:
> Please post your complete stacktrace - the provided one appears
> incomplete/truncated.
>
> On Thu, Sep 19, 2013 at 1:07 AM, Siddhi Mehta <sm...@gmail.com>
> wrote:
> > Hey All,
> >
> > I am running a Hadoop 2.0(cdh4.2.1) cluster.
> >
> > From the job client whenever we try to get the job status from the
> cluster
> > we run into the following IOException.
> > The exception has no message on the cause of the IOException.
> > Any ideas on what could be causing this issue.
> >
> > Looking at the code it seems like it fails while talking to the
> > jobhistoryserver.
> > The application where the job client runs has most of the ports blocked.
> > We opened a the following ports for the job client to talk to the
> > cluster(8030,8031,8032 8033,19888, 10020) but still see the exception.
> > Is there a defined list of ports on which a mapreduce client communicates
> > with the Hadoop Cluster
> >
> > Cause1: java.io.IOException
> > Cause1-StackTrace:
> > at
> >
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
> > at
> >
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385)
> > at
> > org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:487)
> > at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:635)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:633)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > at
> > org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:633)
> > at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:651)
> >
> > Thanks,
> > Siddhi
>
>
>
> --
> Harsh J
>
Re: Yarn Exception while getting JobStatus
Posted by Siddhi Mehta <sm...@gmail.com>.
Hey Harsh,
Here is the more complete stacktrace. I had truncated it earlier since it
was application specific.
Let me know if this helps.
Thrown: java.lang.RuntimeException java.io.IOException
Thrown-StackTrace:
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:37)
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:19)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Cause0: java.lang.RuntimeException java.io.IOException
Cause0-StackTrace:
at
hadoop.client.RemoteHadoopJobClientImpl.getJobStatus(RemoteHadoopJobClientImpl.java:45)
at HadoopJobUtil.updateRunningHadoopJobs(HadoopJobUtil.java:260)
at HadoopJobUpdaterProcess.executeInner(HadoopJobUpdaterProcess.java:54)
at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:35)
... 6 shared with parent
Cause1: java.io.IOException
Cause1-StackTrace:
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:487)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:635)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:633)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:633)
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:651)
at
hadoop.client.RemoteHadoopJobClientImpl.getJobStatus(RemoteHadoopJobClientImpl.java:39)
... 16 shared with parent
On Wed, Sep 18, 2013 at 6:36 PM, Harsh J <ha...@cloudera.com> wrote:
> Please post your complete stacktrace - the provided one appears
> incomplete/truncated.
>
> On Thu, Sep 19, 2013 at 1:07 AM, Siddhi Mehta <sm...@gmail.com>
> wrote:
> > Hey All,
> >
> > I am running a Hadoop 2.0(cdh4.2.1) cluster.
> >
> > From the job client whenever we try to get the job status from the
> cluster
> > we run into the following IOException.
> > The exception has no message on the cause of the IOException.
> > Any ideas on what could be causing this issue.
> >
> > Looking at the code it seems like it fails while talking to the
> > jobhistoryserver.
> > The application where the job client runs has most of the ports blocked.
> > We opened a the following ports for the job client to talk to the
> > cluster(8030,8031,8032 8033,19888, 10020) but still see the exception.
> > Is there a defined list of ports on which a mapreduce client communicates
> > with the Hadoop Cluster
> >
> > Cause1: java.io.IOException
> > Cause1-StackTrace:
> > at
> >
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317)
> > at
> >
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385)
> > at
> > org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:487)
> > at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:635)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:633)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > at
> > org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:633)
> > at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:651)
> >
> > Thanks,
> > Siddhi
>
>
>
> --
> Harsh J
>