You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Karam Singh (JIRA)" <ji...@apache.org> on 2008/04/18 14:08:21 UTC

[jira] Created: (HADOOP-3276) Got org.apache.hadoop.mapred.JobTracker$IllegalStateException while running hadoop job on cluster allocated by hod.

Got org.apache.hadoop.mapred.JobTracker$IllegalStateException while running hadoop job on cluster allocated by hod.
-------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-3276
                 URL: https://issues.apache.org/jira/browse/HADOOP-3276
             Project: Hadoop Core
          Issue Type: Bug
    Affects Versions: 0.17.0
            Reporter: Karam Singh
            Priority: Minor


Allocated cluster using hod of 5 nodes . It returned the both Mapred and Hdfs UI addresses. Also checked out of nodes on 3 taskterakers nodes were using getTaskTrackers() api. 
Then tried to run wordcount example and got following -:
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.mapred.JobTracker$IllegalStateException: Job tracker still initializing
        at org.apache.hadoop.mapred.JobTracker.ensureRunning(JobTracker.java:1722)
        at org.apache.hadoop.mapred.JobTracker.getNewJobId(JobTracker.java:1730)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

        at org.apache.hadoop.ipc.Client.call(Client.java:557)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
        at $Proxy1.getNewJobId(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at $Proxy1.getNewJobId(Unknown Source)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:696)


(Here dfs was static with permissions on.)
On further checking there was problem in static dfs setting, but Web UI was getting started resulting successful allocation, but was not able create mparedsystem directory causing JobTracker$IllegalStateException.
So, It seems if Web UI up then hod assumes that Mapred JT is running
(This is rare condition)


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-3276) Got org.apache.hadoop.mapred.JobTracker$IllegalStateException while running hadoop job on cluster allocated by hod.

Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12593743#action_12593743 ] 

Vinod Kumar Vavilapalli commented on HADOOP-3276:
-------------------------------------------------

Leaving this issue as is, to track any changes to be done in HOD, if and after HADOOP-3289 is resolved.

> Got org.apache.hadoop.mapred.JobTracker$IllegalStateException while running hadoop job on cluster allocated by hod.
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3276
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/hod
>    Affects Versions: 0.17.0
>            Reporter: Karam Singh
>            Priority: Minor
>
> Allocated cluster using hod of 5 nodes . It returned the both Mapred and Hdfs UI addresses. Also checked out of nodes on 3 taskterakers nodes were using getTaskTrackers() api. 
> Then tried to run wordcount example and got following -:
> org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.mapred.JobTracker$IllegalStateException: Job tracker still initializing
>         at org.apache.hadoop.mapred.JobTracker.ensureRunning(JobTracker.java:1722)
>         at org.apache.hadoop.mapred.JobTracker.getNewJobId(JobTracker.java:1730)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>         at org.apache.hadoop.ipc.Client.call(Client.java:557)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>         at $Proxy1.getNewJobId(Unknown Source)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>         at $Proxy1.getNewJobId(Unknown Source)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:696)
> (Here dfs was static with permissions on.)
> On further checking there was problem in static dfs setting, but Web UI was getting started resulting successful allocation, but was not able create mparedsystem directory causing JobTracker$IllegalStateException.
> So, It seems if Web UI up then hod assumes that Mapred JT is running
> (This is rare condition)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-3276) Got org.apache.hadoop.mapred.JobTracker$IllegalStateException while running hadoop job on cluster allocated by hod.

Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590853#action_12590853 ] 

Vinod Kumar Vavilapalli commented on HADOOP-3276:
-------------------------------------------------

This particular problem occurs because of improper setup of Mapred system directory in DFS. But, this can be hit in general when either user submits jobs or HOD continues running script commands, even while JT is still in the state of initialization due to various reasons. HOD currently puts in its best effort to avoid this - ping Mapred Jetty if it is alive - the last thing that can be done to check JT status by HOD from outside without any support in Hadoop. And as the above situation makes it clear, we definitely need support from Hadoop to know when JT can really accepts jobs. Filed a Jira (HADOOP-3289) to address this.

> Got org.apache.hadoop.mapred.JobTracker$IllegalStateException while running hadoop job on cluster allocated by hod.
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3276
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/hod
>    Affects Versions: 0.17.0
>            Reporter: Karam Singh
>            Priority: Minor
>
> Allocated cluster using hod of 5 nodes . It returned the both Mapred and Hdfs UI addresses. Also checked out of nodes on 3 taskterakers nodes were using getTaskTrackers() api. 
> Then tried to run wordcount example and got following -:
> org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.mapred.JobTracker$IllegalStateException: Job tracker still initializing
>         at org.apache.hadoop.mapred.JobTracker.ensureRunning(JobTracker.java:1722)
>         at org.apache.hadoop.mapred.JobTracker.getNewJobId(JobTracker.java:1730)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>         at org.apache.hadoop.ipc.Client.call(Client.java:557)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>         at $Proxy1.getNewJobId(Unknown Source)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>         at $Proxy1.getNewJobId(Unknown Source)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:696)
> (Here dfs was static with permissions on.)
> On further checking there was problem in static dfs setting, but Web UI was getting started resulting successful allocation, but was not able create mparedsystem directory causing JobTracker$IllegalStateException.
> So, It seems if Web UI up then hod assumes that Mapred JT is running
> (This is rare condition)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-3276) Got org.apache.hadoop.mapred.JobTracker$IllegalStateException while running hadoop job on cluster allocated by hod.

Posted by "Karam Singh (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karam Singh updated HADOOP-3276:
--------------------------------

    Component/s: contrib/hod

> Got org.apache.hadoop.mapred.JobTracker$IllegalStateException while running hadoop job on cluster allocated by hod.
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3276
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/hod
>    Affects Versions: 0.17.0
>            Reporter: Karam Singh
>            Priority: Minor
>
> Allocated cluster using hod of 5 nodes . It returned the both Mapred and Hdfs UI addresses. Also checked out of nodes on 3 taskterakers nodes were using getTaskTrackers() api. 
> Then tried to run wordcount example and got following -:
> org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.mapred.JobTracker$IllegalStateException: Job tracker still initializing
>         at org.apache.hadoop.mapred.JobTracker.ensureRunning(JobTracker.java:1722)
>         at org.apache.hadoop.mapred.JobTracker.getNewJobId(JobTracker.java:1730)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>         at org.apache.hadoop.ipc.Client.call(Client.java:557)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>         at $Proxy1.getNewJobId(Unknown Source)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>         at $Proxy1.getNewJobId(Unknown Source)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:696)
> (Here dfs was static with permissions on.)
> On further checking there was problem in static dfs setting, but Web UI was getting started resulting successful allocation, but was not able create mparedsystem directory causing JobTracker$IllegalStateException.
> So, It seems if Web UI up then hod assumes that Mapred JT is running
> (This is rare condition)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.