You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Karim Awara <ka...@kaust.edu.sa> on 2014/04/17 15:23:08 UTC

Problem with jobtracker hadoop 1.2

Hi,

Whenever I start the hadoop on 24 machines, the following exception happens
on the jobtracker log file on the namenode:   I would appreciate any help.
Thank you.



2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
safe mode to false. Requested by : karim
2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory: Creating
DONE folder at file:/home/karim/systems/hadoop-1.1.2/logs/history/done
2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
server being initialized in embedded mode
2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
Started job history server at: localhost:50030
2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
History Server web address: localhost:50030
2014-04-17 16:16:31,449 INFO
org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
inactive
2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
jobtracker.info could only be replicated to 0 nodes, instead of 1
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

    at org.apache.hadoop.ipc.Client.call(Client.java:1107)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
    at $Proxy7.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
    at $Proxy7.addBlock(Unknown Source)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)

2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block null bad datanode[0] nodes == null
2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
get block locations. Source file
"/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
jobtracker.info" - Aborting...
2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
to file
hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
jobtracker.info failed!
2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
FileSystem is not ready yet!
2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed to
initialize recovery manager.
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
jobtracker.info could only be replicated to 0 nodes, instead of 1
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

--
Best Regards,
Karim Ahmed Awara

-- 

------------------------------
This message and its contents, including attachments are intended solely 
for the original recipient. If you are not the intended recipient or have 
received this message in error, please notify me immediately and delete 
this message from your computer system. Any unauthorized use or 
distribution is prohibited. Please consider the environment before printing 
this email.

Re: Problem with jobtracker hadoop 1.2

Posted by Shumin Guo <gs...@gmail.com>.
It seems you are using the local FS rather than HDFS. You need to make sure
your hdfs cluster is up and running.


On Thu, Apr 17, 2014 at 6:42 PM, Shengjun Xin <sx...@gopivotal.com> wrote:

> Did you start datanode service?
>
>
> On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> Whenever I start the hadoop on 24 machines, the following exception
>> happens on the jobtracker log file on the namenode:   I would appreciate
>> any help. Thank you.
>>
>>
>>
>> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
>> safe mode to false. Requested by : karim
>> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker:
>> Cleaning up the system directory
>> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory:
>> Creating DONE folder at
>> file:/home/karim/systems/hadoop-1.1.2/logs/history/done
>> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
>> server being initialized in embedded mode
>> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
>> Started job history server at: localhost:50030
>> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
>> History Server web address: localhost:50030
>> 2014-04-17 16:16:31,449 INFO
>> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
>> inactive
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>>
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
>> get block locations. Source file
>> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info" - Aborting...
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
>> to file
>> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info failed!
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
>> FileSystem is not ready yet!
>> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
>> to initialize recovery manager.
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
>
> --
> Regards
> Shengjun
>

Re: Problem with jobtracker hadoop 1.2

Posted by Shumin Guo <gs...@gmail.com>.
It seems you are using the local FS rather than HDFS. You need to make sure
your hdfs cluster is up and running.


On Thu, Apr 17, 2014 at 6:42 PM, Shengjun Xin <sx...@gopivotal.com> wrote:

> Did you start datanode service?
>
>
> On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> Whenever I start the hadoop on 24 machines, the following exception
>> happens on the jobtracker log file on the namenode:   I would appreciate
>> any help. Thank you.
>>
>>
>>
>> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
>> safe mode to false. Requested by : karim
>> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker:
>> Cleaning up the system directory
>> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory:
>> Creating DONE folder at
>> file:/home/karim/systems/hadoop-1.1.2/logs/history/done
>> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
>> server being initialized in embedded mode
>> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
>> Started job history server at: localhost:50030
>> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
>> History Server web address: localhost:50030
>> 2014-04-17 16:16:31,449 INFO
>> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
>> inactive
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>>
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
>> get block locations. Source file
>> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info" - Aborting...
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
>> to file
>> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info failed!
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
>> FileSystem is not ready yet!
>> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
>> to initialize recovery manager.
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
>
> --
> Regards
> Shengjun
>

Re: Problem with jobtracker hadoop 1.2

Posted by Azuryy Yu <az...@gmail.com>.
There is because your HDFS has no space left. please check your datanodes
are all started. also please check dfs.datanode.du.reserved in
hdfs-site.xml to make sure you don't reserve large capacity.


On Fri, Apr 18, 2014 at 7:42 AM, Shengjun Xin <sx...@gopivotal.com> wrote:

> Did you start datanode service?
>
>
> On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> Whenever I start the hadoop on 24 machines, the following exception
>> happens on the jobtracker log file on the namenode:   I would appreciate
>> any help. Thank you.
>>
>>
>>
>> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
>> safe mode to false. Requested by : karim
>> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker:
>> Cleaning up the system directory
>> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory:
>> Creating DONE folder at
>> file:/home/karim/systems/hadoop-1.1.2/logs/history/done
>> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
>> server being initialized in embedded mode
>> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
>> Started job history server at: localhost:50030
>> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
>> History Server web address: localhost:50030
>> 2014-04-17 16:16:31,449 INFO
>> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
>> inactive
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>>
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
>> get block locations. Source file
>> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info" - Aborting...
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
>> to file
>> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info failed!
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
>> FileSystem is not ready yet!
>> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
>> to initialize recovery manager.
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
>
> --
> Regards
> Shengjun
>

Re: Problem with jobtracker hadoop 1.2

Posted by Azuryy Yu <az...@gmail.com>.
There is because your HDFS has no space left. please check your datanodes
are all started. also please check dfs.datanode.du.reserved in
hdfs-site.xml to make sure you don't reserve large capacity.


On Fri, Apr 18, 2014 at 7:42 AM, Shengjun Xin <sx...@gopivotal.com> wrote:

> Did you start datanode service?
>
>
> On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> Whenever I start the hadoop on 24 machines, the following exception
>> happens on the jobtracker log file on the namenode:   I would appreciate
>> any help. Thank you.
>>
>>
>>
>> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
>> safe mode to false. Requested by : karim
>> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker:
>> Cleaning up the system directory
>> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory:
>> Creating DONE folder at
>> file:/home/karim/systems/hadoop-1.1.2/logs/history/done
>> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
>> server being initialized in embedded mode
>> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
>> Started job history server at: localhost:50030
>> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
>> History Server web address: localhost:50030
>> 2014-04-17 16:16:31,449 INFO
>> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
>> inactive
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>>
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
>> get block locations. Source file
>> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info" - Aborting...
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
>> to file
>> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info failed!
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
>> FileSystem is not ready yet!
>> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
>> to initialize recovery manager.
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
>
> --
> Regards
> Shengjun
>

Re: Problem with jobtracker hadoop 1.2

Posted by Shumin Guo <gs...@gmail.com>.
It seems you are using the local FS rather than HDFS. You need to make sure
your hdfs cluster is up and running.


On Thu, Apr 17, 2014 at 6:42 PM, Shengjun Xin <sx...@gopivotal.com> wrote:

> Did you start datanode service?
>
>
> On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> Whenever I start the hadoop on 24 machines, the following exception
>> happens on the jobtracker log file on the namenode:   I would appreciate
>> any help. Thank you.
>>
>>
>>
>> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
>> safe mode to false. Requested by : karim
>> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker:
>> Cleaning up the system directory
>> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory:
>> Creating DONE folder at
>> file:/home/karim/systems/hadoop-1.1.2/logs/history/done
>> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
>> server being initialized in embedded mode
>> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
>> Started job history server at: localhost:50030
>> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
>> History Server web address: localhost:50030
>> 2014-04-17 16:16:31,449 INFO
>> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
>> inactive
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>>
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
>> get block locations. Source file
>> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info" - Aborting...
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
>> to file
>> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info failed!
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
>> FileSystem is not ready yet!
>> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
>> to initialize recovery manager.
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
>
> --
> Regards
> Shengjun
>

Re: Problem with jobtracker hadoop 1.2

Posted by Shumin Guo <gs...@gmail.com>.
It seems you are using the local FS rather than HDFS. You need to make sure
your hdfs cluster is up and running.


On Thu, Apr 17, 2014 at 6:42 PM, Shengjun Xin <sx...@gopivotal.com> wrote:

> Did you start datanode service?
>
>
> On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> Whenever I start the hadoop on 24 machines, the following exception
>> happens on the jobtracker log file on the namenode:   I would appreciate
>> any help. Thank you.
>>
>>
>>
>> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
>> safe mode to false. Requested by : karim
>> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker:
>> Cleaning up the system directory
>> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory:
>> Creating DONE folder at
>> file:/home/karim/systems/hadoop-1.1.2/logs/history/done
>> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
>> server being initialized in embedded mode
>> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
>> Started job history server at: localhost:50030
>> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
>> History Server web address: localhost:50030
>> 2014-04-17 16:16:31,449 INFO
>> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
>> inactive
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>>
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
>> get block locations. Source file
>> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info" - Aborting...
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
>> to file
>> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info failed!
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
>> FileSystem is not ready yet!
>> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
>> to initialize recovery manager.
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
>
> --
> Regards
> Shengjun
>

Re: Problem with jobtracker hadoop 1.2

Posted by Azuryy Yu <az...@gmail.com>.
There is because your HDFS has no space left. please check your datanodes
are all started. also please check dfs.datanode.du.reserved in
hdfs-site.xml to make sure you don't reserve large capacity.


On Fri, Apr 18, 2014 at 7:42 AM, Shengjun Xin <sx...@gopivotal.com> wrote:

> Did you start datanode service?
>
>
> On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> Whenever I start the hadoop on 24 machines, the following exception
>> happens on the jobtracker log file on the namenode:   I would appreciate
>> any help. Thank you.
>>
>>
>>
>> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
>> safe mode to false. Requested by : karim
>> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker:
>> Cleaning up the system directory
>> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory:
>> Creating DONE folder at
>> file:/home/karim/systems/hadoop-1.1.2/logs/history/done
>> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
>> server being initialized in embedded mode
>> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
>> Started job history server at: localhost:50030
>> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
>> History Server web address: localhost:50030
>> 2014-04-17 16:16:31,449 INFO
>> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
>> inactive
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>>
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
>> get block locations. Source file
>> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info" - Aborting...
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
>> to file
>> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info failed!
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
>> FileSystem is not ready yet!
>> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
>> to initialize recovery manager.
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
>
> --
> Regards
> Shengjun
>

Re: Problem with jobtracker hadoop 1.2

Posted by Azuryy Yu <az...@gmail.com>.
There is because your HDFS has no space left. please check your datanodes
are all started. also please check dfs.datanode.du.reserved in
hdfs-site.xml to make sure you don't reserve large capacity.


On Fri, Apr 18, 2014 at 7:42 AM, Shengjun Xin <sx...@gopivotal.com> wrote:

> Did you start datanode service?
>
>
> On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:
>
>> Hi,
>>
>> Whenever I start the hadoop on 24 machines, the following exception
>> happens on the jobtracker log file on the namenode:   I would appreciate
>> any help. Thank you.
>>
>>
>>
>> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
>> safe mode to false. Requested by : karim
>> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker:
>> Cleaning up the system directory
>> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory:
>> Creating DONE folder at
>> file:/home/karim/systems/hadoop-1.1.2/logs/history/done
>> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
>> server being initialized in embedded mode
>> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
>> Started job history server at: localhost:50030
>> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
>> History Server web address: localhost:50030
>> 2014-04-17 16:16:31,449 INFO
>> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
>> inactive
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>>     at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>>     at $Proxy7.addBlock(Unknown Source)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>>     at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>>
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
>> get block locations. Source file
>> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info" - Aborting...
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
>> to file
>> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info failed!
>> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
>> FileSystem is not ready yet!
>> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
>> to initialize recovery manager.
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
>> jobtracker.info could only be replicated to 0 nodes, instead of 1
>>     at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> --
>> Best Regards,
>> Karim Ahmed Awara
>>
>> ------------------------------
>> This message and its contents, including attachments are intended solely
>> for the original recipient. If you are not the intended recipient or have
>> received this message in error, please notify me immediately and delete
>> this message from your computer system. Any unauthorized use or
>> distribution is prohibited. Please consider the environment before printing
>> this email.
>
>
>
>
> --
> Regards
> Shengjun
>

Re: Problem with jobtracker hadoop 1.2

Posted by Shengjun Xin <sx...@gopivotal.com>.
Did you start datanode service?


On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:

> Hi,
>
> Whenever I start the hadoop on 24 machines, the following exception
> happens on the jobtracker log file on the namenode:   I would appreciate
> any help. Thank you.
>
>
>
> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
> safe mode to false. Requested by : karim
> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
> Loaded the native-hadoop library
> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory: Creating
> DONE folder at file:/home/karim/systems/hadoop-1.1.2/logs/history/done
> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
> server being initialized in embedded mode
> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
> Started job history server at: localhost:50030
> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
> History Server web address: localhost:50030
> 2014-04-17 16:16:31,449 INFO
> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
> inactive
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: File
> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info could only be replicated to 0 nodes, instead of 1
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>     at $Proxy7.addBlock(Unknown Source)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>     at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>     at $Proxy7.addBlock(Unknown Source)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block null bad datanode[0] nodes == null
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
> get block locations. Source file
> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info" - Aborting...
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
> to file
> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info failed!
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
> FileSystem is not ready yet!
> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
> to initialize recovery manager.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info could only be replicated to 0 nodes, instead of 1
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> --
> Best Regards,
> Karim Ahmed Awara
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.




-- 
Regards
Shengjun

Re: Problem with jobtracker hadoop 1.2

Posted by Shengjun Xin <sx...@gopivotal.com>.
Did you start datanode service?


On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:

> Hi,
>
> Whenever I start the hadoop on 24 machines, the following exception
> happens on the jobtracker log file on the namenode:   I would appreciate
> any help. Thank you.
>
>
>
> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
> safe mode to false. Requested by : karim
> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
> Loaded the native-hadoop library
> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory: Creating
> DONE folder at file:/home/karim/systems/hadoop-1.1.2/logs/history/done
> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
> server being initialized in embedded mode
> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
> Started job history server at: localhost:50030
> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
> History Server web address: localhost:50030
> 2014-04-17 16:16:31,449 INFO
> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
> inactive
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: File
> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info could only be replicated to 0 nodes, instead of 1
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>     at $Proxy7.addBlock(Unknown Source)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>     at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>     at $Proxy7.addBlock(Unknown Source)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block null bad datanode[0] nodes == null
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
> get block locations. Source file
> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info" - Aborting...
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
> to file
> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info failed!
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
> FileSystem is not ready yet!
> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
> to initialize recovery manager.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info could only be replicated to 0 nodes, instead of 1
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> --
> Best Regards,
> Karim Ahmed Awara
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.




-- 
Regards
Shengjun

Re: Problem with jobtracker hadoop 1.2

Posted by Shengjun Xin <sx...@gopivotal.com>.
Did you start datanode service?


On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:

> Hi,
>
> Whenever I start the hadoop on 24 machines, the following exception
> happens on the jobtracker log file on the namenode:   I would appreciate
> any help. Thank you.
>
>
>
> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
> safe mode to false. Requested by : karim
> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
> Loaded the native-hadoop library
> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory: Creating
> DONE folder at file:/home/karim/systems/hadoop-1.1.2/logs/history/done
> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
> server being initialized in embedded mode
> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
> Started job history server at: localhost:50030
> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
> History Server web address: localhost:50030
> 2014-04-17 16:16:31,449 INFO
> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
> inactive
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: File
> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info could only be replicated to 0 nodes, instead of 1
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>     at $Proxy7.addBlock(Unknown Source)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>     at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>     at $Proxy7.addBlock(Unknown Source)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block null bad datanode[0] nodes == null
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
> get block locations. Source file
> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info" - Aborting...
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
> to file
> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info failed!
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
> FileSystem is not ready yet!
> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
> to initialize recovery manager.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info could only be replicated to 0 nodes, instead of 1
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> --
> Best Regards,
> Karim Ahmed Awara
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.




-- 
Regards
Shengjun

Re: Problem with jobtracker hadoop 1.2

Posted by Shengjun Xin <sx...@gopivotal.com>.
Did you start datanode service?


On Thu, Apr 17, 2014 at 9:23 PM, Karim Awara <ka...@kaust.edu.sa>wrote:

> Hi,
>
> Whenever I start the hadoop on 24 machines, the following exception
> happens on the jobtracker log file on the namenode:   I would appreciate
> any help. Thank you.
>
>
>
> 2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
> safe mode to false. Requested by : karim
> 2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
> Loaded the native-hadoop library
> 2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory: Creating
> DONE folder at file:/home/karim/systems/hadoop-1.1.2/logs/history/done
> 2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
> server being initialized in embedded mode
> 2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
> Started job history server at: localhost:50030
> 2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
> History Server web address: localhost:50030
> 2014-04-17 16:16:31,449 INFO
> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
> inactive
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient:
> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: File
> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info could only be replicated to 0 nodes, instead of 1
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>     at $Proxy7.addBlock(Unknown Source)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>     at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>     at $Proxy7.addBlock(Unknown Source)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
>
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block null bad datanode[0] nodes == null
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
> get block locations. Source file
> "/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info" - Aborting...
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
> to file
> hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info failed!
> 2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
> FileSystem is not ready yet!
> 2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed
> to initialize recovery manager.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
> jobtracker.info could only be replicated to 0 nodes, instead of 1
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> --
> Best Regards,
> Karim Ahmed Awara
>
> ------------------------------
> This message and its contents, including attachments are intended solely
> for the original recipient. If you are not the intended recipient or have
> received this message in error, please notify me immediately and delete
> this message from your computer system. Any unauthorized use or
> distribution is prohibited. Please consider the environment before printing
> this email.




-- 
Regards
Shengjun