You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by siddharth mathur <si...@gmail.com> on 2013/07/09 18:56:02 UTC
Issues Running Hadoop 1.1.2 on multi-node cluster
Hi,
I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching
this tutorial *
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
*
When I startup the hadoop, I get the folloing error in *all* the
tasktrackers.
"
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051203_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051611_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:* Failed
to get system directory*...
2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
"
*But everything looks fine in the webUI. *
When I run a job, I get the following error but the job completes anyways.
I have* attached the* *screenshots* of the maptask failed error log in the
UI.
*"*
13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process
: 2
13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
attempt_201307091215_0001_m_000001_0, Status : FAILED
Error initializing attempt_201307091215_0001_m_000001_0:
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
at
org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
at
org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at
org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
at java.lang.Thread.run(Thread.java:724)
13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
attempt_201307091215_0001_m_000000_1, Status : FAILED
Error initializing attempt_201307091215_0001_m_000000_1:
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
at
org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
at
org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at
org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
at java.lang.Thread.run(Thread.java:724)
13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001
13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
reduces waiting after reserving slots (ms)=0
13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps
waiting after reserving slots (ms)=0
13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
bytes=117
13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
(bytes)=602996736
13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
snapshot=520691712
13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=3025334272
13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
*"*
Following is the complete tasktracker log on one node after starting the
hadoop (*not running the job*):
*"*
2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = dmkd-1/172.31.21.10
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.1.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
************************************************************/
2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2013-07-09 12:15:15,823 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats registered.
2013-07-09 12:15:15,824 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2013-07-09 12:15:15,824 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
system started
2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2013-07-09 12:15:16,223 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
registered.
2013-07-09 12:15:16,230 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
exists!
2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
global filtersafety
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting
tasktracker with owner as hduser
2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
mapred local directories are: /app/hadoop/tmp/mapred/local
2013-07-09 12:15:21,652 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
registered.
2013-07-09 12:15:21,653 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
TaskTrackerMetrics registered.
2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
SocketReader
2013-07-09 12:15:21,683 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
RpcDetailedActivityForPort36783 registered.
2013-07-09 12:15:21,684 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
RpcActivityForPort36783 registered.
2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 36783: starting
2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 36783: starting
2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 36783: starting
2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 36783: starting
2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 36783: starting
2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker up at: 127.0.0.1/127.0.0.1:36783
2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting
tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting
thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
127.0.0.1/127.0.0.1:36783
2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
exited with exit code 0
2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
disabled.
2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
IndexCache created with max memory = 10485760
2013-07-09 12:15:21,785 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
ShuffleServerMetrics registered.
2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50060
2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50060
webServer.getConnectors()[0].getLocalPort() returned 50060
2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50060
2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50060
2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
FILE_CACHE_SIZE for mapOutputServlet set to : 2000
2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0003 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051627_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051203_0002 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051135_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0002 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051327_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051135_0005 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307050127_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307050127_0002 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051500_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051634_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051203_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051611_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0003 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0003 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051627_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051203_0002 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051135_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0002 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051327_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051135_0005 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307050127_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307050127_0002 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051500_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051634_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051203_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051611_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
*"*
Please help.
Thanks and Regards,
Siddharth
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Sree K <qu...@yahoo.com>.
Siddharth,
The error msgs pointing to file system issues. Make sure that the file system locations you specified in the config files are accurate and accessible.
-Sreedhar
>________________________________
> From: siddharth mathur <si...@gmail.com>
>To: user@hadoop.apache.org
>Sent: Tuesday, July 9, 2013 9:56 AM
>Subject: Issues Running Hadoop 1.1.2 on multi-node cluster
>
>
>
>Hi,
>
>I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching this tutorial http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>
>When I startup the hadoop, I get the folloing error in all the tasktrackers.
>
>
>"
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:Failed to get system directory...
>2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>"
>
>But everything looks fine in the webUI.
>
>
>When I run a job, I get the following error but the job completes anyways. I haveattached the screenshots of the maptask failed error log in the UI.
>
>"
>
>13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process : 2
>13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library
>13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
>13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
>13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>13/07/09 12:29:41 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000001_0, Status : FAILED
>Error initializing attempt_201307091215_0001_m_000001_0:
>ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
>13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>13/07/09 12:30:38 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000000_1, Status : FAILED
>Error initializing attempt_201307091215_0001_m_000000_1:
>ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
>13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001
>13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
>13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
>13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized bytes=117
>13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage (bytes)=602996736
>13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes) snapshot=520691712
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes) snapshot=3025334272
>13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
>"
>
>
>Following is the complete tasktracker log on one node after starting the hadoop (not running the job):
>
>
>"
>
>2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
>/************************************************************
>STARTUP_MSG: Starting TaskTracker
>STARTUP_MSG: host = dmkd-1/172.31.21.10
>STARTUP_MSG: args = []
>STARTUP_MSG: version = 1.1.2
>STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>************************************************************/
>2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
>2013-07-09 12:15:15,823 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
>2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
>2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
>2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
>2013-07-09 12:15:16,223 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
>2013-07-09 12:15:16,230 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
>2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
>2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hduser
>2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /app/hadoop/tmp/mapred/local
>2013-07-09 12:15:21,652 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
>2013-07-09 12:15:21,653 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
>2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
>2013-07-09 12:15:21,683 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort36783 registered.
>2013-07-09 12:15:21,684 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort36783 registered.
>2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 36783: starting
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 36783: starting
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
>2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
>2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
>2013-07-09 12:15:21,785 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
>2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
>2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
>2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
>2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>2013-07-09 12:15:22,286 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
>2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>
>"
>
>
>Please help.
>
>
>
>Thanks and Regards,
>
>Siddharth
>
>
>
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Sree K <qu...@yahoo.com>.
Siddharth,
The error msgs pointing to file system issues. Make sure that the file system locations you specified in the config files are accurate and accessible.
-Sreedhar
>________________________________
> From: siddharth mathur <si...@gmail.com>
>To: user@hadoop.apache.org
>Sent: Tuesday, July 9, 2013 9:56 AM
>Subject: Issues Running Hadoop 1.1.2 on multi-node cluster
>
>
>
>Hi,
>
>I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching this tutorial http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>
>When I startup the hadoop, I get the folloing error in all the tasktrackers.
>
>
>"
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:Failed to get system directory...
>2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>"
>
>But everything looks fine in the webUI.
>
>
>When I run a job, I get the following error but the job completes anyways. I haveattached the screenshots of the maptask failed error log in the UI.
>
>"
>
>13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process : 2
>13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library
>13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
>13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
>13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>13/07/09 12:29:41 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000001_0, Status : FAILED
>Error initializing attempt_201307091215_0001_m_000001_0:
>ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
>13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>13/07/09 12:30:38 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000000_1, Status : FAILED
>Error initializing attempt_201307091215_0001_m_000000_1:
>ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
>13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001
>13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
>13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
>13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized bytes=117
>13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage (bytes)=602996736
>13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes) snapshot=520691712
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes) snapshot=3025334272
>13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
>"
>
>
>Following is the complete tasktracker log on one node after starting the hadoop (not running the job):
>
>
>"
>
>2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
>/************************************************************
>STARTUP_MSG: Starting TaskTracker
>STARTUP_MSG: host = dmkd-1/172.31.21.10
>STARTUP_MSG: args = []
>STARTUP_MSG: version = 1.1.2
>STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>************************************************************/
>2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
>2013-07-09 12:15:15,823 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
>2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
>2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
>2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
>2013-07-09 12:15:16,223 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
>2013-07-09 12:15:16,230 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
>2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
>2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hduser
>2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /app/hadoop/tmp/mapred/local
>2013-07-09 12:15:21,652 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
>2013-07-09 12:15:21,653 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
>2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
>2013-07-09 12:15:21,683 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort36783 registered.
>2013-07-09 12:15:21,684 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort36783 registered.
>2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 36783: starting
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 36783: starting
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
>2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
>2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
>2013-07-09 12:15:21,785 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
>2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
>2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
>2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
>2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>2013-07-09 12:15:22,286 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
>2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>
>"
>
>
>Please help.
>
>
>
>Thanks and Regards,
>
>Siddharth
>
>
>
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Sree K <qu...@yahoo.com>.
Siddharth,
The error msgs pointing to file system issues. Make sure that the file system locations you specified in the config files are accurate and accessible.
-Sreedhar
>________________________________
> From: siddharth mathur <si...@gmail.com>
>To: user@hadoop.apache.org
>Sent: Tuesday, July 9, 2013 9:56 AM
>Subject: Issues Running Hadoop 1.1.2 on multi-node cluster
>
>
>
>Hi,
>
>I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching this tutorial http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>
>When I startup the hadoop, I get the folloing error in all the tasktrackers.
>
>
>"
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:Failed to get system directory...
>2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>"
>
>But everything looks fine in the webUI.
>
>
>When I run a job, I get the following error but the job completes anyways. I haveattached the screenshots of the maptask failed error log in the UI.
>
>"
>
>13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process : 2
>13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library
>13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
>13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
>13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>13/07/09 12:29:41 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000001_0, Status : FAILED
>Error initializing attempt_201307091215_0001_m_000001_0:
>ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
>13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>13/07/09 12:30:38 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000000_1, Status : FAILED
>Error initializing attempt_201307091215_0001_m_000000_1:
>ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
>13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001
>13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
>13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
>13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized bytes=117
>13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage (bytes)=602996736
>13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes) snapshot=520691712
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes) snapshot=3025334272
>13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
>"
>
>
>Following is the complete tasktracker log on one node after starting the hadoop (not running the job):
>
>
>"
>
>2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
>/************************************************************
>STARTUP_MSG: Starting TaskTracker
>STARTUP_MSG: host = dmkd-1/172.31.21.10
>STARTUP_MSG: args = []
>STARTUP_MSG: version = 1.1.2
>STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>************************************************************/
>2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
>2013-07-09 12:15:15,823 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
>2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
>2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
>2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
>2013-07-09 12:15:16,223 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
>2013-07-09 12:15:16,230 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
>2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
>2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hduser
>2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /app/hadoop/tmp/mapred/local
>2013-07-09 12:15:21,652 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
>2013-07-09 12:15:21,653 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
>2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
>2013-07-09 12:15:21,683 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort36783 registered.
>2013-07-09 12:15:21,684 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort36783 registered.
>2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 36783: starting
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 36783: starting
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
>2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
>2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
>2013-07-09 12:15:21,785 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
>2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
>2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
>2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
>2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>2013-07-09 12:15:22,286 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
>2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>
>"
>
>
>Please help.
>
>
>
>Thanks and Regards,
>
>Siddharth
>
>
>
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by siddharth mathur <si...@gmail.com>.
I figured out the issue!
The problem was in the permission to rum Hadoop scripts from root user. I
create a dedicated hadoop user to rum hadoop cluster but one of the time i
accidentally started hadoop from root. Hence, some of the permissions of
hadoop scripts changed.
The solution is to again change the ownership of the hadoop folder to the
dedicated user using chown. Its working fine now.
Thanks a lot for the pointers!
Regards,
Siddharth
On Thu, Jul 11, 2013 at 1:43 AM, Ram <pr...@gmail.com> wrote:
> Hi,
> Please check all directories/files are existed in local system
> configured mapres-site.xml and permissions to the files/directories as
> mapred as user and hadoop as a group.
>
> Hi,
>
>
>
> From,
> P.Ramesh Babu,
> +91-7893442722.
>
>
>
> On Wed, Jul 10, 2013 at 9:36 PM, Leonid Fedotov <lf...@hortonworks.com>wrote:
>
>> Make sure your mapred.local.dir (check it in mapred-site.xml) is actually
>> exists and writable by your mapreduce usewr.
>>
>> *Thank you!*
>> *
>> *
>> *Sincerely,*
>> *Leonid Fedotov*
>>
>>
>> On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
>>
>> Hi Siddharth,
>>
>> While running the multi-node we need to take care of the local host of
>> the slave machine from the error messages the task tracker root directory
>> not able to get to the masters. Please check and rerun it.
>>
>> Thanks,
>> Kiran
>>
>>
>> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
>>> watching this tutorial *
>>> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>>> *
>>>
>>> When I startup the hadoop, I get the folloing error in *all* the
>>> tasktrackers.
>>>
>>> "
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
>>> *...
>>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> "
>>>
>>> *But everything looks fine in the webUI. *
>>>
>>> When I run a job, I get the following error but the job completes
>>> anyways. I have* attached the* *screenshots* of the maptask failed
>>> error log in the UI.
>>>
>>> *"*
>>> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to
>>> process : 2
>>> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
>>> library
>>> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not
>>> loaded
>>> 13/07/09 12:29:37 INFO mapred.JobClient: Running job:
>>> job_201307091215_0001
>>> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>>> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
>>> attempt_201307091215_0001_m_000001_0, Status : FAILED
>>> Error initializing attempt_201307091215_0001_m_000001_0:
>>> ENOENT: No such file or directory
>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>> at
>>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>>> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>>> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>>> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
>>> attempt_201307091215_0001_m_000000_1, Status : FAILED
>>> Error initializing attempt_201307091215_0001_m_000000_1:
>>> ENOENT: No such file or directory
>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>> at
>>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>>> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>>> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
>>> job_201307091215_0001
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>>> reduces waiting after reserving slots (ms)=0
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>>> maps waiting after reserving slots (ms)=0
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>>> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>>> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
>>> bytes=117
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
>>> (bytes)=602996736
>>> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
>>> snapshot=520691712
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
>>> snapshot=3025334272
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>>>
>>> *"*
>>>
>>>
>>> Following is the complete tasktracker log on one node after starting the
>>> hadoop (*not running the job*):
>>>
>>> *"*
>>> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
>>> STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting TaskTracker
>>> STARTUP_MSG: host = dmkd-1/172.31.21.10
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.1.2
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
>>> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>>> ************************************************************/
>>> 2013-07-09 12:15:15,722 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-07-09 12:15:15,823 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-07-09 12:15:15,824 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-07-09 12:15:15,824 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>>> system started
>>> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
>>> Loaded the native-hadoop library
>>> 2013-07-09 12:15:16,223 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-07-09 12:15:16,230 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> org.mortbay.log.Slf4jLog
>>> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
>>> global filtersafety
>>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>>> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting tasktracker with owner as hduser
>>> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
>>> mapred local directories are: /app/hadoop/tmp/mapred/local
>>> 2013-07-09 12:15:21,652 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
>>> registered.
>>> 2013-07-09 12:15:21,653 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> TaskTrackerMetrics registered.
>>> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
>>> SocketReader
>>> 2013-07-09 12:15:21,683 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> RpcDetailedActivityForPort36783 registered.
>>> 2013-07-09 12:15:21,684 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> RpcActivityForPort36783 registered.
>>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 0 on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 1 on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 2 on 36783: starting
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 3 on 36783: starting
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>>> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
>>> 127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
>>> exited with exit code 0
>>> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Using ResourceCalculatorPlugin :
>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>>> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
>>> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
>>> disabled.
>>> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
>>> IndexCache created with max memory = 10485760
>>> 2013-07-09 12:15:21,785 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> ShuffleServerMetrics registered.
>>> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
>>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>>> -1. Opening the listener on 50060
>>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
>>> listener.getLocalPort() returned 50060
>>> webServer.getConnectors()[0].getLocalPort() returned 50060
>>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
>>> bound to port 50060
>>> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>>> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
>>> SelectChannelConnector@0.0.0.0:50060
>>> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
>>> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051627_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051327_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0005 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051500_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051634_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051627_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051327_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0005 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051500_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051634_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>>
>>> *"*
>>>
>>> Please help.
>>>
>>>
>>> Thanks and Regards,
>>> Siddharth
>>>
>>>
>>>
>>>
>>
>>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by siddharth mathur <si...@gmail.com>.
I figured out the issue!
The problem was in the permission to rum Hadoop scripts from root user. I
create a dedicated hadoop user to rum hadoop cluster but one of the time i
accidentally started hadoop from root. Hence, some of the permissions of
hadoop scripts changed.
The solution is to again change the ownership of the hadoop folder to the
dedicated user using chown. Its working fine now.
Thanks a lot for the pointers!
Regards,
Siddharth
On Thu, Jul 11, 2013 at 1:43 AM, Ram <pr...@gmail.com> wrote:
> Hi,
> Please check all directories/files are existed in local system
> configured mapres-site.xml and permissions to the files/directories as
> mapred as user and hadoop as a group.
>
> Hi,
>
>
>
> From,
> P.Ramesh Babu,
> +91-7893442722.
>
>
>
> On Wed, Jul 10, 2013 at 9:36 PM, Leonid Fedotov <lf...@hortonworks.com>wrote:
>
>> Make sure your mapred.local.dir (check it in mapred-site.xml) is actually
>> exists and writable by your mapreduce usewr.
>>
>> *Thank you!*
>> *
>> *
>> *Sincerely,*
>> *Leonid Fedotov*
>>
>>
>> On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
>>
>> Hi Siddharth,
>>
>> While running the multi-node we need to take care of the local host of
>> the slave machine from the error messages the task tracker root directory
>> not able to get to the masters. Please check and rerun it.
>>
>> Thanks,
>> Kiran
>>
>>
>> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
>>> watching this tutorial *
>>> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>>> *
>>>
>>> When I startup the hadoop, I get the folloing error in *all* the
>>> tasktrackers.
>>>
>>> "
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
>>> *...
>>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> "
>>>
>>> *But everything looks fine in the webUI. *
>>>
>>> When I run a job, I get the following error but the job completes
>>> anyways. I have* attached the* *screenshots* of the maptask failed
>>> error log in the UI.
>>>
>>> *"*
>>> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to
>>> process : 2
>>> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
>>> library
>>> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not
>>> loaded
>>> 13/07/09 12:29:37 INFO mapred.JobClient: Running job:
>>> job_201307091215_0001
>>> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>>> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
>>> attempt_201307091215_0001_m_000001_0, Status : FAILED
>>> Error initializing attempt_201307091215_0001_m_000001_0:
>>> ENOENT: No such file or directory
>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>> at
>>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>>> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>>> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>>> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
>>> attempt_201307091215_0001_m_000000_1, Status : FAILED
>>> Error initializing attempt_201307091215_0001_m_000000_1:
>>> ENOENT: No such file or directory
>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>> at
>>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>>> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>>> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
>>> job_201307091215_0001
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>>> reduces waiting after reserving slots (ms)=0
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>>> maps waiting after reserving slots (ms)=0
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>>> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>>> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
>>> bytes=117
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
>>> (bytes)=602996736
>>> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
>>> snapshot=520691712
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
>>> snapshot=3025334272
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>>>
>>> *"*
>>>
>>>
>>> Following is the complete tasktracker log on one node after starting the
>>> hadoop (*not running the job*):
>>>
>>> *"*
>>> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
>>> STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting TaskTracker
>>> STARTUP_MSG: host = dmkd-1/172.31.21.10
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.1.2
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
>>> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>>> ************************************************************/
>>> 2013-07-09 12:15:15,722 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-07-09 12:15:15,823 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-07-09 12:15:15,824 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-07-09 12:15:15,824 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>>> system started
>>> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
>>> Loaded the native-hadoop library
>>> 2013-07-09 12:15:16,223 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-07-09 12:15:16,230 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> org.mortbay.log.Slf4jLog
>>> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
>>> global filtersafety
>>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>>> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting tasktracker with owner as hduser
>>> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
>>> mapred local directories are: /app/hadoop/tmp/mapred/local
>>> 2013-07-09 12:15:21,652 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
>>> registered.
>>> 2013-07-09 12:15:21,653 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> TaskTrackerMetrics registered.
>>> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
>>> SocketReader
>>> 2013-07-09 12:15:21,683 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> RpcDetailedActivityForPort36783 registered.
>>> 2013-07-09 12:15:21,684 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> RpcActivityForPort36783 registered.
>>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 0 on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 1 on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 2 on 36783: starting
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 3 on 36783: starting
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>>> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
>>> 127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
>>> exited with exit code 0
>>> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Using ResourceCalculatorPlugin :
>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>>> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
>>> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
>>> disabled.
>>> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
>>> IndexCache created with max memory = 10485760
>>> 2013-07-09 12:15:21,785 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> ShuffleServerMetrics registered.
>>> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
>>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>>> -1. Opening the listener on 50060
>>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
>>> listener.getLocalPort() returned 50060
>>> webServer.getConnectors()[0].getLocalPort() returned 50060
>>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
>>> bound to port 50060
>>> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>>> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
>>> SelectChannelConnector@0.0.0.0:50060
>>> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
>>> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051627_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051327_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0005 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051500_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051634_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051627_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051327_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0005 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051500_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051634_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>>
>>> *"*
>>>
>>> Please help.
>>>
>>>
>>> Thanks and Regards,
>>> Siddharth
>>>
>>>
>>>
>>>
>>
>>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by siddharth mathur <si...@gmail.com>.
I figured out the issue!
The problem was in the permission to rum Hadoop scripts from root user. I
create a dedicated hadoop user to rum hadoop cluster but one of the time i
accidentally started hadoop from root. Hence, some of the permissions of
hadoop scripts changed.
The solution is to again change the ownership of the hadoop folder to the
dedicated user using chown. Its working fine now.
Thanks a lot for the pointers!
Regards,
Siddharth
On Thu, Jul 11, 2013 at 1:43 AM, Ram <pr...@gmail.com> wrote:
> Hi,
> Please check all directories/files are existed in local system
> configured mapres-site.xml and permissions to the files/directories as
> mapred as user and hadoop as a group.
>
> Hi,
>
>
>
> From,
> P.Ramesh Babu,
> +91-7893442722.
>
>
>
> On Wed, Jul 10, 2013 at 9:36 PM, Leonid Fedotov <lf...@hortonworks.com>wrote:
>
>> Make sure your mapred.local.dir (check it in mapred-site.xml) is actually
>> exists and writable by your mapreduce usewr.
>>
>> *Thank you!*
>> *
>> *
>> *Sincerely,*
>> *Leonid Fedotov*
>>
>>
>> On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
>>
>> Hi Siddharth,
>>
>> While running the multi-node we need to take care of the local host of
>> the slave machine from the error messages the task tracker root directory
>> not able to get to the masters. Please check and rerun it.
>>
>> Thanks,
>> Kiran
>>
>>
>> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
>>> watching this tutorial *
>>> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>>> *
>>>
>>> When I startup the hadoop, I get the folloing error in *all* the
>>> tasktrackers.
>>>
>>> "
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
>>> *...
>>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> "
>>>
>>> *But everything looks fine in the webUI. *
>>>
>>> When I run a job, I get the following error but the job completes
>>> anyways. I have* attached the* *screenshots* of the maptask failed
>>> error log in the UI.
>>>
>>> *"*
>>> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to
>>> process : 2
>>> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
>>> library
>>> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not
>>> loaded
>>> 13/07/09 12:29:37 INFO mapred.JobClient: Running job:
>>> job_201307091215_0001
>>> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>>> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
>>> attempt_201307091215_0001_m_000001_0, Status : FAILED
>>> Error initializing attempt_201307091215_0001_m_000001_0:
>>> ENOENT: No such file or directory
>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>> at
>>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>>> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>>> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>>> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
>>> attempt_201307091215_0001_m_000000_1, Status : FAILED
>>> Error initializing attempt_201307091215_0001_m_000000_1:
>>> ENOENT: No such file or directory
>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>> at
>>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>>> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>>> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
>>> job_201307091215_0001
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>>> reduces waiting after reserving slots (ms)=0
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>>> maps waiting after reserving slots (ms)=0
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>>> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>>> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
>>> bytes=117
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
>>> (bytes)=602996736
>>> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
>>> snapshot=520691712
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
>>> snapshot=3025334272
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>>>
>>> *"*
>>>
>>>
>>> Following is the complete tasktracker log on one node after starting the
>>> hadoop (*not running the job*):
>>>
>>> *"*
>>> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
>>> STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting TaskTracker
>>> STARTUP_MSG: host = dmkd-1/172.31.21.10
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.1.2
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
>>> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>>> ************************************************************/
>>> 2013-07-09 12:15:15,722 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-07-09 12:15:15,823 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-07-09 12:15:15,824 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-07-09 12:15:15,824 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>>> system started
>>> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
>>> Loaded the native-hadoop library
>>> 2013-07-09 12:15:16,223 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-07-09 12:15:16,230 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> org.mortbay.log.Slf4jLog
>>> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
>>> global filtersafety
>>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>>> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting tasktracker with owner as hduser
>>> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
>>> mapred local directories are: /app/hadoop/tmp/mapred/local
>>> 2013-07-09 12:15:21,652 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
>>> registered.
>>> 2013-07-09 12:15:21,653 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> TaskTrackerMetrics registered.
>>> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
>>> SocketReader
>>> 2013-07-09 12:15:21,683 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> RpcDetailedActivityForPort36783 registered.
>>> 2013-07-09 12:15:21,684 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> RpcActivityForPort36783 registered.
>>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 0 on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 1 on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 2 on 36783: starting
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 3 on 36783: starting
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>>> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
>>> 127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
>>> exited with exit code 0
>>> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Using ResourceCalculatorPlugin :
>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>>> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
>>> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
>>> disabled.
>>> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
>>> IndexCache created with max memory = 10485760
>>> 2013-07-09 12:15:21,785 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> ShuffleServerMetrics registered.
>>> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
>>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>>> -1. Opening the listener on 50060
>>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
>>> listener.getLocalPort() returned 50060
>>> webServer.getConnectors()[0].getLocalPort() returned 50060
>>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
>>> bound to port 50060
>>> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>>> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
>>> SelectChannelConnector@0.0.0.0:50060
>>> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
>>> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051627_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051327_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0005 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051500_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051634_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051627_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051327_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0005 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051500_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051634_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>>
>>> *"*
>>>
>>> Please help.
>>>
>>>
>>> Thanks and Regards,
>>> Siddharth
>>>
>>>
>>>
>>>
>>
>>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by siddharth mathur <si...@gmail.com>.
I figured out the issue!
The problem was in the permission to rum Hadoop scripts from root user. I
create a dedicated hadoop user to rum hadoop cluster but one of the time i
accidentally started hadoop from root. Hence, some of the permissions of
hadoop scripts changed.
The solution is to again change the ownership of the hadoop folder to the
dedicated user using chown. Its working fine now.
Thanks a lot for the pointers!
Regards,
Siddharth
On Thu, Jul 11, 2013 at 1:43 AM, Ram <pr...@gmail.com> wrote:
> Hi,
> Please check all directories/files are existed in local system
> configured mapres-site.xml and permissions to the files/directories as
> mapred as user and hadoop as a group.
>
> Hi,
>
>
>
> From,
> P.Ramesh Babu,
> +91-7893442722.
>
>
>
> On Wed, Jul 10, 2013 at 9:36 PM, Leonid Fedotov <lf...@hortonworks.com>wrote:
>
>> Make sure your mapred.local.dir (check it in mapred-site.xml) is actually
>> exists and writable by your mapreduce usewr.
>>
>> *Thank you!*
>> *
>> *
>> *Sincerely,*
>> *Leonid Fedotov*
>>
>>
>> On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
>>
>> Hi Siddharth,
>>
>> While running the multi-node we need to take care of the local host of
>> the slave machine from the error messages the task tracker root directory
>> not able to get to the masters. Please check and rerun it.
>>
>> Thanks,
>> Kiran
>>
>>
>> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
>>> watching this tutorial *
>>> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>>> *
>>>
>>> When I startup the hadoop, I get the folloing error in *all* the
>>> tasktrackers.
>>>
>>> "
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
>>> *...
>>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> "
>>>
>>> *But everything looks fine in the webUI. *
>>>
>>> When I run a job, I get the following error but the job completes
>>> anyways. I have* attached the* *screenshots* of the maptask failed
>>> error log in the UI.
>>>
>>> *"*
>>> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to
>>> process : 2
>>> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
>>> library
>>> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not
>>> loaded
>>> 13/07/09 12:29:37 INFO mapred.JobClient: Running job:
>>> job_201307091215_0001
>>> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>>> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
>>> attempt_201307091215_0001_m_000001_0, Status : FAILED
>>> Error initializing attempt_201307091215_0001_m_000001_0:
>>> ENOENT: No such file or directory
>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>> at
>>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>>> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>>> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>>> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
>>> attempt_201307091215_0001_m_000000_1, Status : FAILED
>>> Error initializing attempt_201307091215_0001_m_000000_1:
>>> ENOENT: No such file or directory
>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>> at
>>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>>> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>>> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
>>> job_201307091215_0001
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>>> reduces waiting after reserving slots (ms)=0
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>>> maps waiting after reserving slots (ms)=0
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>>> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>>> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
>>> bytes=117
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
>>> (bytes)=602996736
>>> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
>>> snapshot=520691712
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
>>> snapshot=3025334272
>>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>>>
>>> *"*
>>>
>>>
>>> Following is the complete tasktracker log on one node after starting the
>>> hadoop (*not running the job*):
>>>
>>> *"*
>>> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
>>> STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting TaskTracker
>>> STARTUP_MSG: host = dmkd-1/172.31.21.10
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.1.2
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
>>> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>>> ************************************************************/
>>> 2013-07-09 12:15:15,722 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-07-09 12:15:15,823 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-07-09 12:15:15,824 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-07-09 12:15:15,824 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>>> system started
>>> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
>>> Loaded the native-hadoop library
>>> 2013-07-09 12:15:16,223 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-07-09 12:15:16,230 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> org.mortbay.log.Slf4jLog
>>> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
>>> global filtersafety
>>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>>> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting tasktracker with owner as hduser
>>> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
>>> mapred local directories are: /app/hadoop/tmp/mapred/local
>>> 2013-07-09 12:15:21,652 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
>>> registered.
>>> 2013-07-09 12:15:21,653 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> TaskTrackerMetrics registered.
>>> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
>>> SocketReader
>>> 2013-07-09 12:15:21,683 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> RpcDetailedActivityForPort36783 registered.
>>> 2013-07-09 12:15:21,684 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> RpcActivityForPort36783 registered.
>>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 0 on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 1 on 36783: starting
>>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 2 on 36783: starting
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 3 on 36783: starting
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>>> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
>>> 127.0.0.1/127.0.0.1:36783
>>> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
>>> exited with exit code 0
>>> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Using ResourceCalculatorPlugin :
>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>>> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
>>> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
>>> disabled.
>>> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
>>> IndexCache created with max memory = 10485760
>>> 2013-07-09 12:15:21,785 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> ShuffleServerMetrics registered.
>>> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
>>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>>> -1. Opening the listener on 50060
>>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
>>> listener.getLocalPort() returned 50060
>>> webServer.getConnectors()[0].getLocalPort() returned 50060
>>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
>>> bound to port 50060
>>> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>>> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
>>> SelectChannelConnector@0.0.0.0:50060
>>> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
>>> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051627_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051327_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0005 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0002 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051500_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051634_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921773
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0003 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051627_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051327_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051135_0005 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307050127_0002 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051500_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051634_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307032034_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051203_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>>> Adding job_201307051611_0001 for user-log deletion with
>>> retainTimeStamp:1373472921775
>>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Failed to get system directory...
>>>
>>> *"*
>>>
>>> Please help.
>>>
>>>
>>> Thanks and Regards,
>>> Siddharth
>>>
>>>
>>>
>>>
>>
>>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Ram <pr...@gmail.com>.
Hi,
Please check all directories/files are existed in local system
configured mapres-site.xml and permissions to the files/directories as
mapred as user and hadoop as a group.
Hi,
From,
P.Ramesh Babu,
+91-7893442722.
On Wed, Jul 10, 2013 at 9:36 PM, Leonid Fedotov <lf...@hortonworks.com>wrote:
> Make sure your mapred.local.dir (check it in mapred-site.xml) is actually
> exists and writable by your mapreduce usewr.
>
> *Thank you!*
> *
> *
> *Sincerely,*
> *Leonid Fedotov*
>
>
> On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
>
> Hi Siddharth,
>
> While running the multi-node we need to take care of the local host of the
> slave machine from the error messages the task tracker root directory not
> able to get to the masters. Please check and rerun it.
>
> Thanks,
> Kiran
>
>
> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
>
>> Hi,
>>
>> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
>> watching this tutorial *
>> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>> *
>>
>> When I startup the hadoop, I get the folloing error in *all* the
>> tasktrackers.
>>
>> "
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
>> *...
>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> "
>>
>> *But everything looks fine in the webUI. *
>>
>> When I run a job, I get the following error but the job completes
>> anyways. I have* attached the* *screenshots* of the maptask failed error
>> log in the UI.
>>
>> *"*
>> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to
>> process : 2
>> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/09 12:29:37 INFO mapred.JobClient: Running job:
>> job_201307091215_0001
>> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
>> attempt_201307091215_0001_m_000001_0, Status : FAILED
>> Error initializing attempt_201307091215_0001_m_000001_0:
>> ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>> at
>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at
>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>> at
>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>> at java.lang.Thread.run(Thread.java:724)
>>
>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
>> attempt_201307091215_0001_m_000000_1, Status : FAILED
>> Error initializing attempt_201307091215_0001_m_000000_1:
>> ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>> at
>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at
>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>> at
>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>> at java.lang.Thread.run(Thread.java:724)
>>
>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
>> job_201307091215_0001
>> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
>> bytes=117
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=602996736
>> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=520691712
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=3025334272
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>>
>> *"*
>>
>>
>> Following is the complete tasktracker log on one node after starting the
>> hadoop (*not running the job*):
>>
>> *"*
>> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
>> STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting TaskTracker
>> STARTUP_MSG: host = dmkd-1/172.31.21.10
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.1.2
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
>> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>> ************************************************************/
>> 2013-07-09 12:15:15,722 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2013-07-09 12:15:15,823 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2013-07-09 12:15:15,824 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-07-09 12:15:15,824 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>> system started
>> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2013-07-09 12:15:16,223 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>> registered.
>> 2013-07-09 12:15:16,230 WARN
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>> exists!
>> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
>> global filtersafety
>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting tasktracker with owner as hduser
>> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
>> mapred local directories are: /app/hadoop/tmp/mapred/local
>> 2013-07-09 12:15:21,652 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
>> registered.
>> 2013-07-09 12:15:21,653 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> TaskTrackerMetrics registered.
>> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
>> SocketReader
>> 2013-07-09 12:15:21,683 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> RpcDetailedActivityForPort36783 registered.
>> 2013-07-09 12:15:21,684 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> RpcActivityForPort36783 registered.
>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 0 on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 1 on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 2 on 36783: starting
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 3 on 36783: starting
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
>> 127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
>> exited with exit code 0
>> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using
>> ResourceCalculatorPlugin :
>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
>> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
>> disabled.
>> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
>> IndexCache created with max memory = 10485760
>> 2013-07-09 12:15:21,785 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> ShuffleServerMetrics registered.
>> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>> -1. Opening the listener on 50060
>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
>> listener.getLocalPort() returned 50060
>> webServer.getConnectors()[0].getLocalPort() returned 50060
>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 50060
>> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
>> SelectChannelConnector@0.0.0.0:50060
>> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
>> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051627_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051327_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0005 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051500_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051634_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051627_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051327_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0005 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051500_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051634_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>>
>> *"*
>>
>> Please help.
>>
>>
>> Thanks and Regards,
>> Siddharth
>>
>>
>>
>>
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Ram <pr...@gmail.com>.
Hi,
Please check all directories/files are existed in local system
configured mapres-site.xml and permissions to the files/directories as
mapred as user and hadoop as a group.
Hi,
From,
P.Ramesh Babu,
+91-7893442722.
On Wed, Jul 10, 2013 at 9:36 PM, Leonid Fedotov <lf...@hortonworks.com>wrote:
> Make sure your mapred.local.dir (check it in mapred-site.xml) is actually
> exists and writable by your mapreduce usewr.
>
> *Thank you!*
> *
> *
> *Sincerely,*
> *Leonid Fedotov*
>
>
> On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
>
> Hi Siddharth,
>
> While running the multi-node we need to take care of the local host of the
> slave machine from the error messages the task tracker root directory not
> able to get to the masters. Please check and rerun it.
>
> Thanks,
> Kiran
>
>
> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
>
>> Hi,
>>
>> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
>> watching this tutorial *
>> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>> *
>>
>> When I startup the hadoop, I get the folloing error in *all* the
>> tasktrackers.
>>
>> "
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
>> *...
>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> "
>>
>> *But everything looks fine in the webUI. *
>>
>> When I run a job, I get the following error but the job completes
>> anyways. I have* attached the* *screenshots* of the maptask failed error
>> log in the UI.
>>
>> *"*
>> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to
>> process : 2
>> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/09 12:29:37 INFO mapred.JobClient: Running job:
>> job_201307091215_0001
>> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
>> attempt_201307091215_0001_m_000001_0, Status : FAILED
>> Error initializing attempt_201307091215_0001_m_000001_0:
>> ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>> at
>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at
>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>> at
>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>> at java.lang.Thread.run(Thread.java:724)
>>
>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
>> attempt_201307091215_0001_m_000000_1, Status : FAILED
>> Error initializing attempt_201307091215_0001_m_000000_1:
>> ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>> at
>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at
>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>> at
>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>> at java.lang.Thread.run(Thread.java:724)
>>
>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
>> job_201307091215_0001
>> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
>> bytes=117
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=602996736
>> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=520691712
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=3025334272
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>>
>> *"*
>>
>>
>> Following is the complete tasktracker log on one node after starting the
>> hadoop (*not running the job*):
>>
>> *"*
>> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
>> STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting TaskTracker
>> STARTUP_MSG: host = dmkd-1/172.31.21.10
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.1.2
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
>> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>> ************************************************************/
>> 2013-07-09 12:15:15,722 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2013-07-09 12:15:15,823 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2013-07-09 12:15:15,824 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-07-09 12:15:15,824 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>> system started
>> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2013-07-09 12:15:16,223 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>> registered.
>> 2013-07-09 12:15:16,230 WARN
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>> exists!
>> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
>> global filtersafety
>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting tasktracker with owner as hduser
>> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
>> mapred local directories are: /app/hadoop/tmp/mapred/local
>> 2013-07-09 12:15:21,652 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
>> registered.
>> 2013-07-09 12:15:21,653 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> TaskTrackerMetrics registered.
>> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
>> SocketReader
>> 2013-07-09 12:15:21,683 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> RpcDetailedActivityForPort36783 registered.
>> 2013-07-09 12:15:21,684 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> RpcActivityForPort36783 registered.
>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 0 on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 1 on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 2 on 36783: starting
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 3 on 36783: starting
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
>> 127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
>> exited with exit code 0
>> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using
>> ResourceCalculatorPlugin :
>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
>> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
>> disabled.
>> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
>> IndexCache created with max memory = 10485760
>> 2013-07-09 12:15:21,785 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> ShuffleServerMetrics registered.
>> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>> -1. Opening the listener on 50060
>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
>> listener.getLocalPort() returned 50060
>> webServer.getConnectors()[0].getLocalPort() returned 50060
>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 50060
>> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
>> SelectChannelConnector@0.0.0.0:50060
>> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
>> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051627_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051327_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0005 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051500_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051634_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051627_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051327_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0005 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051500_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051634_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>>
>> *"*
>>
>> Please help.
>>
>>
>> Thanks and Regards,
>> Siddharth
>>
>>
>>
>>
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Ram <pr...@gmail.com>.
Hi,
Please check all directories/files are existed in local system
configured mapres-site.xml and permissions to the files/directories as
mapred as user and hadoop as a group.
Hi,
From,
P.Ramesh Babu,
+91-7893442722.
On Wed, Jul 10, 2013 at 9:36 PM, Leonid Fedotov <lf...@hortonworks.com>wrote:
> Make sure your mapred.local.dir (check it in mapred-site.xml) is actually
> exists and writable by your mapreduce usewr.
>
> *Thank you!*
> *
> *
> *Sincerely,*
> *Leonid Fedotov*
>
>
> On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
>
> Hi Siddharth,
>
> While running the multi-node we need to take care of the local host of the
> slave machine from the error messages the task tracker root directory not
> able to get to the masters. Please check and rerun it.
>
> Thanks,
> Kiran
>
>
> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
>
>> Hi,
>>
>> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
>> watching this tutorial *
>> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>> *
>>
>> When I startup the hadoop, I get the folloing error in *all* the
>> tasktrackers.
>>
>> "
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
>> *...
>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> "
>>
>> *But everything looks fine in the webUI. *
>>
>> When I run a job, I get the following error but the job completes
>> anyways. I have* attached the* *screenshots* of the maptask failed error
>> log in the UI.
>>
>> *"*
>> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to
>> process : 2
>> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/09 12:29:37 INFO mapred.JobClient: Running job:
>> job_201307091215_0001
>> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
>> attempt_201307091215_0001_m_000001_0, Status : FAILED
>> Error initializing attempt_201307091215_0001_m_000001_0:
>> ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>> at
>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at
>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>> at
>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>> at java.lang.Thread.run(Thread.java:724)
>>
>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
>> attempt_201307091215_0001_m_000000_1, Status : FAILED
>> Error initializing attempt_201307091215_0001_m_000000_1:
>> ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>> at
>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at
>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>> at
>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>> at java.lang.Thread.run(Thread.java:724)
>>
>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
>> job_201307091215_0001
>> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
>> bytes=117
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=602996736
>> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=520691712
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=3025334272
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>>
>> *"*
>>
>>
>> Following is the complete tasktracker log on one node after starting the
>> hadoop (*not running the job*):
>>
>> *"*
>> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
>> STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting TaskTracker
>> STARTUP_MSG: host = dmkd-1/172.31.21.10
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.1.2
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
>> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>> ************************************************************/
>> 2013-07-09 12:15:15,722 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2013-07-09 12:15:15,823 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2013-07-09 12:15:15,824 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-07-09 12:15:15,824 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>> system started
>> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2013-07-09 12:15:16,223 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>> registered.
>> 2013-07-09 12:15:16,230 WARN
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>> exists!
>> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
>> global filtersafety
>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting tasktracker with owner as hduser
>> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
>> mapred local directories are: /app/hadoop/tmp/mapred/local
>> 2013-07-09 12:15:21,652 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
>> registered.
>> 2013-07-09 12:15:21,653 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> TaskTrackerMetrics registered.
>> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
>> SocketReader
>> 2013-07-09 12:15:21,683 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> RpcDetailedActivityForPort36783 registered.
>> 2013-07-09 12:15:21,684 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> RpcActivityForPort36783 registered.
>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 0 on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 1 on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 2 on 36783: starting
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 3 on 36783: starting
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
>> 127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
>> exited with exit code 0
>> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using
>> ResourceCalculatorPlugin :
>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
>> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
>> disabled.
>> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
>> IndexCache created with max memory = 10485760
>> 2013-07-09 12:15:21,785 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> ShuffleServerMetrics registered.
>> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>> -1. Opening the listener on 50060
>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
>> listener.getLocalPort() returned 50060
>> webServer.getConnectors()[0].getLocalPort() returned 50060
>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 50060
>> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
>> SelectChannelConnector@0.0.0.0:50060
>> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
>> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051627_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051327_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0005 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051500_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051634_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051627_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051327_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0005 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051500_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051634_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>>
>> *"*
>>
>> Please help.
>>
>>
>> Thanks and Regards,
>> Siddharth
>>
>>
>>
>>
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Ram <pr...@gmail.com>.
Hi,
Please check all directories/files are existed in local system
configured mapres-site.xml and permissions to the files/directories as
mapred as user and hadoop as a group.
Hi,
From,
P.Ramesh Babu,
+91-7893442722.
On Wed, Jul 10, 2013 at 9:36 PM, Leonid Fedotov <lf...@hortonworks.com>wrote:
> Make sure your mapred.local.dir (check it in mapred-site.xml) is actually
> exists and writable by your mapreduce usewr.
>
> *Thank you!*
> *
> *
> *Sincerely,*
> *Leonid Fedotov*
>
>
> On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
>
> Hi Siddharth,
>
> While running the multi-node we need to take care of the local host of the
> slave machine from the error messages the task tracker root directory not
> able to get to the masters. Please check and rerun it.
>
> Thanks,
> Kiran
>
>
> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
>
>> Hi,
>>
>> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
>> watching this tutorial *
>> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>> *
>>
>> When I startup the hadoop, I get the folloing error in *all* the
>> tasktrackers.
>>
>> "
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
>> *...
>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> "
>>
>> *But everything looks fine in the webUI. *
>>
>> When I run a job, I get the following error but the job completes
>> anyways. I have* attached the* *screenshots* of the maptask failed error
>> log in the UI.
>>
>> *"*
>> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to
>> process : 2
>> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/09 12:29:37 INFO mapred.JobClient: Running job:
>> job_201307091215_0001
>> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
>> attempt_201307091215_0001_m_000001_0, Status : FAILED
>> Error initializing attempt_201307091215_0001_m_000001_0:
>> ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>> at
>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at
>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>> at
>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>> at java.lang.Thread.run(Thread.java:724)
>>
>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
>> attempt_201307091215_0001_m_000000_1, Status : FAILED
>> Error initializing attempt_201307091215_0001_m_000000_1:
>> ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>> at
>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at
>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
>> at
>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
>> at java.lang.Thread.run(Thread.java:724)
>>
>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
>> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
>> job_201307091215_0001
>> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
>> bytes=117
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=602996736
>> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=520691712
>> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=3025334272
>> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>>
>> *"*
>>
>>
>> Following is the complete tasktracker log on one node after starting the
>> hadoop (*not running the job*):
>>
>> *"*
>> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
>> STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting TaskTracker
>> STARTUP_MSG: host = dmkd-1/172.31.21.10
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.1.2
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
>> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>> ************************************************************/
>> 2013-07-09 12:15:15,722 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2013-07-09 12:15:15,823 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2013-07-09 12:15:15,824 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-07-09 12:15:15,824 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>> system started
>> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
>> Loaded the native-hadoop library
>> 2013-07-09 12:15:16,223 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>> registered.
>> 2013-07-09 12:15:16,230 WARN
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>> exists!
>> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
>> global filtersafety
>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting tasktracker with owner as hduser
>> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
>> mapred local directories are: /app/hadoop/tmp/mapred/local
>> 2013-07-09 12:15:21,652 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
>> registered.
>> 2013-07-09 12:15:21,653 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> TaskTrackerMetrics registered.
>> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
>> SocketReader
>> 2013-07-09 12:15:21,683 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> RpcDetailedActivityForPort36783 registered.
>> 2013-07-09 12:15:21,684 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> RpcActivityForPort36783 registered.
>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 0 on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 1 on 36783: starting
>> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 2 on 36783: starting
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 3 on 36783: starting
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
>> 127.0.0.1/127.0.0.1:36783
>> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
>> exited with exit code 0
>> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using
>> ResourceCalculatorPlugin :
>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
>> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
>> disabled.
>> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
>> IndexCache created with max memory = 10485760
>> 2013-07-09 12:15:21,785 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> ShuffleServerMetrics registered.
>> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>> -1. Opening the listener on 50060
>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
>> listener.getLocalPort() returned 50060
>> webServer.getConnectors()[0].getLocalPort() returned 50060
>> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 50060
>> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
>> SelectChannelConnector@0.0.0.0:50060
>> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
>> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051627_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051327_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0005 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0002 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051500_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051634_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921773
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0003 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051627_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051327_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051135_0005 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307050127_0002 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051500_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051634_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307032034_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051203_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
>> Adding job_201307051611_0001 for user-log deletion with
>> retainTimeStamp:1373472921775
>> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed
>> to get system directory...
>>
>> *"*
>>
>> Please help.
>>
>>
>> Thanks and Regards,
>> Siddharth
>>
>>
>>
>>
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Leonid Fedotov <lf...@hortonworks.com>.
Make sure your mapred.local.dir (check it in mapred-site.xml) is actually exists and writable by your mapreduce usewr.
Thank you!
Sincerely,
Leonid Fedotov
On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
> Hi Siddharth,
>
> While running the multi-node we need to take care of the local host of the slave machine from the error messages the task tracker root directory not able to get to the masters. Please check and rerun it.
>
> Thanks,
> Kiran
>
>
> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching this tutorial http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>
> When I startup the hadoop, I get the folloing error in all the tasktrackers.
>
> "
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> "
>
> But everything looks fine in the webUI.
>
> When I run a job, I get the following error but the job completes anyways. I have attached the screenshots of the maptask failed error log in the UI.
>
> "
> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process : 2
> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000001_0, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000001_0:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000000_1, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000000_1:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001
> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage (bytes)=602996736
> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes) snapshot=520691712
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes) snapshot=3025334272
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
> "
>
>
> Following is the complete tasktracker log on one node after starting the hadoop (not running the job):
>
> "
> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = dmkd-1/172.31.21.10
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.1.2
> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
> ************************************************************/
> 2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
> 2013-07-09 12:15:15,823 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
> 2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
> 2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
> 2013-07-09 12:15:16,223 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
> 2013-07-09 12:15:16,230 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hduser
> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /app/hadoop/tmp/mapred/local
> 2013-07-09 12:15:21,652 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
> 2013-07-09 12:15:21,653 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
> 2013-07-09 12:15:21,683 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort36783 registered.
> 2013-07-09 12:15:21,684 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort36783 registered.
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
> 2013-07-09 12:15:21,785 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>
> "
>
> Please help.
>
>
> Thanks and Regards,
> Siddharth
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Leonid Fedotov <lf...@hortonworks.com>.
Make sure your mapred.local.dir (check it in mapred-site.xml) is actually exists and writable by your mapreduce usewr.
Thank you!
Sincerely,
Leonid Fedotov
On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
> Hi Siddharth,
>
> While running the multi-node we need to take care of the local host of the slave machine from the error messages the task tracker root directory not able to get to the masters. Please check and rerun it.
>
> Thanks,
> Kiran
>
>
> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching this tutorial http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>
> When I startup the hadoop, I get the folloing error in all the tasktrackers.
>
> "
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> "
>
> But everything looks fine in the webUI.
>
> When I run a job, I get the following error but the job completes anyways. I have attached the screenshots of the maptask failed error log in the UI.
>
> "
> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process : 2
> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000001_0, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000001_0:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000000_1, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000000_1:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001
> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage (bytes)=602996736
> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes) snapshot=520691712
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes) snapshot=3025334272
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
> "
>
>
> Following is the complete tasktracker log on one node after starting the hadoop (not running the job):
>
> "
> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = dmkd-1/172.31.21.10
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.1.2
> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
> ************************************************************/
> 2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
> 2013-07-09 12:15:15,823 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
> 2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
> 2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
> 2013-07-09 12:15:16,223 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
> 2013-07-09 12:15:16,230 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hduser
> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /app/hadoop/tmp/mapred/local
> 2013-07-09 12:15:21,652 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
> 2013-07-09 12:15:21,653 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
> 2013-07-09 12:15:21,683 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort36783 registered.
> 2013-07-09 12:15:21,684 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort36783 registered.
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
> 2013-07-09 12:15:21,785 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>
> "
>
> Please help.
>
>
> Thanks and Regards,
> Siddharth
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Leonid Fedotov <lf...@hortonworks.com>.
Make sure your mapred.local.dir (check it in mapred-site.xml) is actually exists and writable by your mapreduce usewr.
Thank you!
Sincerely,
Leonid Fedotov
On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
> Hi Siddharth,
>
> While running the multi-node we need to take care of the local host of the slave machine from the error messages the task tracker root directory not able to get to the masters. Please check and rerun it.
>
> Thanks,
> Kiran
>
>
> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching this tutorial http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>
> When I startup the hadoop, I get the folloing error in all the tasktrackers.
>
> "
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> "
>
> But everything looks fine in the webUI.
>
> When I run a job, I get the following error but the job completes anyways. I have attached the screenshots of the maptask failed error log in the UI.
>
> "
> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process : 2
> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000001_0, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000001_0:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000000_1, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000000_1:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001
> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage (bytes)=602996736
> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes) snapshot=520691712
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes) snapshot=3025334272
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
> "
>
>
> Following is the complete tasktracker log on one node after starting the hadoop (not running the job):
>
> "
> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = dmkd-1/172.31.21.10
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.1.2
> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
> ************************************************************/
> 2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
> 2013-07-09 12:15:15,823 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
> 2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
> 2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
> 2013-07-09 12:15:16,223 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
> 2013-07-09 12:15:16,230 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hduser
> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /app/hadoop/tmp/mapred/local
> 2013-07-09 12:15:21,652 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
> 2013-07-09 12:15:21,653 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
> 2013-07-09 12:15:21,683 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort36783 registered.
> 2013-07-09 12:15:21,684 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort36783 registered.
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
> 2013-07-09 12:15:21,785 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>
> "
>
> Please help.
>
>
> Thanks and Regards,
> Siddharth
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Leonid Fedotov <lf...@hortonworks.com>.
Make sure your mapred.local.dir (check it in mapred-site.xml) is actually exists and writable by your mapreduce usewr.
Thank you!
Sincerely,
Leonid Fedotov
On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote:
> Hi Siddharth,
>
> While running the multi-node we need to take care of the local host of the slave machine from the error messages the task tracker root directory not able to get to the masters. Please check and rerun it.
>
> Thanks,
> Kiran
>
>
> On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching this tutorial http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>
> When I startup the hadoop, I get the folloing error in all the tasktrackers.
>
> "
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> "
>
> But everything looks fine in the webUI.
>
> When I run a job, I get the following error but the job completes anyways. I have attached the screenshots of the maptask failed error log in the UI.
>
> "
> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process : 2
> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000001_0, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000001_0:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000000_1, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000000_1:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001
> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage (bytes)=602996736
> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes) snapshot=520691712
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes) snapshot=3025334272
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
> "
>
>
> Following is the complete tasktracker log on one node after starting the hadoop (not running the job):
>
> "
> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = dmkd-1/172.31.21.10
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.1.2
> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
> ************************************************************/
> 2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
> 2013-07-09 12:15:15,823 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
> 2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
> 2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
> 2013-07-09 12:15:16,223 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
> 2013-07-09 12:15:16,230 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hduser
> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /app/hadoop/tmp/mapred/local
> 2013-07-09 12:15:21,652 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
> 2013-07-09 12:15:21,653 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
> 2013-07-09 12:15:21,683 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort36783 registered.
> 2013-07-09 12:15:21,684 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort36783 registered.
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
> 2013-07-09 12:15:21,785 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>
> "
>
> Please help.
>
>
> Thanks and Regards,
> Siddharth
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Kiran Dangeti <ki...@gmail.com>.
Hi Siddharth,
While running the multi-node we need to take care of the local host of the
slave machine from the error messages the task tracker root directory not
able to get to the masters. Please check and rerun it.
Thanks,
Kiran
On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
> Hi,
>
> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
> watching this tutorial *
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
> *
>
> When I startup the hadoop, I get the folloing error in *all* the
> tasktrackers.
>
> "
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
> *...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> "
>
> *But everything looks fine in the webUI. *
>
> When I run a job, I get the following error but the job completes anyways.
> I have* attached the* *screenshots* of the maptask failed error log in
> the UI.
>
> *"*
> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process
> : 2
> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
> attempt_201307091215_0001_m_000001_0, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000001_0:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at
> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at
> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
> attempt_201307091215_0001_m_000000_1, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000000_1:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at
> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at
> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
> job_201307091215_0001
> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
> bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
> (bytes)=602996736
> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=520691712
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=3025334272
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
> *"*
>
>
> Following is the complete tasktracker log on one node after starting the
> hadoop (*not running the job*):
>
> *"*
> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = dmkd-1/172.31.21.10
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.1.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
> ************************************************************/
> 2013-07-09 12:15:15,722 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-07-09 12:15:15,823 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-07-09 12:15:15,824 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-07-09 12:15:15,824 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> system started
> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
> Loaded the native-hadoop library
> 2013-07-09 12:15:16,223 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-07-09 12:15:16,230 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
> global filtersafety
> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tasktracker with owner as hduser
> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
> mapred local directories are: /app/hadoop/tmp/mapred/local
> 2013-07-09 12:15:21,652 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
> registered.
> 2013-07-09 12:15:21,653 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> TaskTrackerMetrics registered.
> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
> SocketReader
> 2013-07-09 12:15:21,683 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> RpcDetailedActivityForPort36783 registered.
> 2013-07-09 12:15:21,684 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> RpcActivityForPort36783 registered.
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
> 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
> exited with exit code 0
> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> disabled.
> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache created with max memory = 10485760
> 2013-07-09 12:15:21,785 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> ShuffleServerMetrics registered.
> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50060
> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051627_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051327_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0005 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051500_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051634_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051627_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051327_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0005 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051500_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051634_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
>
> *"*
>
> Please help.
>
>
> Thanks and Regards,
> Siddharth
>
>
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Sree K <qu...@yahoo.com>.
Siddharth,
The error msgs pointing to file system issues. Make sure that the file system locations you specified in the config files are accurate and accessible.
-Sreedhar
>________________________________
> From: siddharth mathur <si...@gmail.com>
>To: user@hadoop.apache.org
>Sent: Tuesday, July 9, 2013 9:56 AM
>Subject: Issues Running Hadoop 1.1.2 on multi-node cluster
>
>
>
>Hi,
>
>I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching this tutorial http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>
>When I startup the hadoop, I get the folloing error in all the tasktrackers.
>
>
>"
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:Failed to get system directory...
>2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>"
>
>But everything looks fine in the webUI.
>
>
>When I run a job, I get the following error but the job completes anyways. I haveattached the screenshots of the maptask failed error log in the UI.
>
>"
>
>13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process : 2
>13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library
>13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
>13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
>13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
>13/07/09 12:29:41 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000001_0, Status : FAILED
>Error initializing attempt_201307091215_0001_m_000001_0:
>ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
>13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
>13/07/09 12:29:41 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
>13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
>13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
>13/07/09 12:30:38 INFO mapred.JobClient: Task Id : attempt_201307091215_0001_m_000000_1, Status : FAILED
>Error initializing attempt_201307091215_0001_m_000000_1:
>ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
>13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
>13/07/09 12:30:38 WARN mapred.JobClient: Error reading task outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
>13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
>13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
>13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001
>13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
>13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
>13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
>13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
>13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
>13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
>13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
>13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
>13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
>13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
>13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
>13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
>13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
>13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
>13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
>13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
>13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
>13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
>13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized bytes=117
>13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
>13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
>13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
>13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage (bytes)=602996736
>13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
>13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
>13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
>13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
>13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes) snapshot=520691712
>13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
>13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes) snapshot=3025334272
>13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
>"
>
>
>Following is the complete tasktracker log on one node after starting the hadoop (not running the job):
>
>
>"
>
>2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
>/************************************************************
>STARTUP_MSG: Starting TaskTracker
>STARTUP_MSG: host = dmkd-1/172.31.21.10
>STARTUP_MSG: args = []
>STARTUP_MSG: version = 1.1.2
>STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
>************************************************************/
>2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
>2013-07-09 12:15:15,823 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
>2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
>2013-07-09 12:15:15,824 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
>2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
>2013-07-09 12:15:16,223 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
>2013-07-09 12:15:16,230 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
>2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
>2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hduser
>2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /app/hadoop/tmp/mapred/local
>2013-07-09 12:15:21,652 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
>2013-07-09 12:15:21,653 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
>2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
>2013-07-09 12:15:21,683 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort36783 registered.
>2013-07-09 12:15:21,684 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort36783 registered.
>2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 36783: starting
>2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 36783: starting
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 36783: starting
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: 127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
>2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
>2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
>2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
>2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
>2013-07-09 12:15:21,785 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
>2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
>2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
>2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
>2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
>2013-07-09 12:15:22,286 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
>2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921773
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775
>2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory...
>
>"
>
>
>Please help.
>
>
>
>Thanks and Regards,
>
>Siddharth
>
>
>
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Kiran Dangeti <ki...@gmail.com>.
Hi Siddharth,
While running the multi-node we need to take care of the local host of the
slave machine from the error messages the task tracker root directory not
able to get to the masters. Please check and rerun it.
Thanks,
Kiran
On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
> Hi,
>
> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
> watching this tutorial *
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
> *
>
> When I startup the hadoop, I get the folloing error in *all* the
> tasktrackers.
>
> "
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
> *...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> "
>
> *But everything looks fine in the webUI. *
>
> When I run a job, I get the following error but the job completes anyways.
> I have* attached the* *screenshots* of the maptask failed error log in
> the UI.
>
> *"*
> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process
> : 2
> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
> attempt_201307091215_0001_m_000001_0, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000001_0:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at
> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at
> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
> attempt_201307091215_0001_m_000000_1, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000000_1:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at
> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at
> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
> job_201307091215_0001
> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
> bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
> (bytes)=602996736
> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=520691712
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=3025334272
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
> *"*
>
>
> Following is the complete tasktracker log on one node after starting the
> hadoop (*not running the job*):
>
> *"*
> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = dmkd-1/172.31.21.10
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.1.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
> ************************************************************/
> 2013-07-09 12:15:15,722 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-07-09 12:15:15,823 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-07-09 12:15:15,824 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-07-09 12:15:15,824 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> system started
> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
> Loaded the native-hadoop library
> 2013-07-09 12:15:16,223 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-07-09 12:15:16,230 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
> global filtersafety
> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tasktracker with owner as hduser
> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
> mapred local directories are: /app/hadoop/tmp/mapred/local
> 2013-07-09 12:15:21,652 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
> registered.
> 2013-07-09 12:15:21,653 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> TaskTrackerMetrics registered.
> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
> SocketReader
> 2013-07-09 12:15:21,683 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> RpcDetailedActivityForPort36783 registered.
> 2013-07-09 12:15:21,684 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> RpcActivityForPort36783 registered.
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
> 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
> exited with exit code 0
> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> disabled.
> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache created with max memory = 10485760
> 2013-07-09 12:15:21,785 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> ShuffleServerMetrics registered.
> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50060
> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051627_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051327_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0005 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051500_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051634_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051627_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051327_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0005 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051500_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051634_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
>
> *"*
>
> Please help.
>
>
> Thanks and Regards,
> Siddharth
>
>
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Kiran Dangeti <ki...@gmail.com>.
Hi Siddharth,
While running the multi-node we need to take care of the local host of the
slave machine from the error messages the task tracker root directory not
able to get to the masters. Please check and rerun it.
Thanks,
Kiran
On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
> Hi,
>
> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
> watching this tutorial *
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
> *
>
> When I startup the hadoop, I get the folloing error in *all* the
> tasktrackers.
>
> "
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
> *...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> "
>
> *But everything looks fine in the webUI. *
>
> When I run a job, I get the following error but the job completes anyways.
> I have* attached the* *screenshots* of the maptask failed error log in
> the UI.
>
> *"*
> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process
> : 2
> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
> attempt_201307091215_0001_m_000001_0, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000001_0:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at
> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at
> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
> attempt_201307091215_0001_m_000000_1, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000000_1:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at
> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at
> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
> job_201307091215_0001
> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
> bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
> (bytes)=602996736
> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=520691712
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=3025334272
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
> *"*
>
>
> Following is the complete tasktracker log on one node after starting the
> hadoop (*not running the job*):
>
> *"*
> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = dmkd-1/172.31.21.10
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.1.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
> ************************************************************/
> 2013-07-09 12:15:15,722 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-07-09 12:15:15,823 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-07-09 12:15:15,824 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-07-09 12:15:15,824 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> system started
> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
> Loaded the native-hadoop library
> 2013-07-09 12:15:16,223 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-07-09 12:15:16,230 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
> global filtersafety
> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tasktracker with owner as hduser
> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
> mapred local directories are: /app/hadoop/tmp/mapred/local
> 2013-07-09 12:15:21,652 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
> registered.
> 2013-07-09 12:15:21,653 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> TaskTrackerMetrics registered.
> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
> SocketReader
> 2013-07-09 12:15:21,683 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> RpcDetailedActivityForPort36783 registered.
> 2013-07-09 12:15:21,684 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> RpcActivityForPort36783 registered.
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
> 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
> exited with exit code 0
> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> disabled.
> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache created with max memory = 10485760
> 2013-07-09 12:15:21,785 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> ShuffleServerMetrics registered.
> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50060
> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051627_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051327_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0005 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051500_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051634_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051627_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051327_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0005 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051500_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051634_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
>
> *"*
>
> Please help.
>
>
> Thanks and Regards,
> Siddharth
>
>
>
>
Re: Issues Running Hadoop 1.1.2 on multi-node cluster
Posted by Kiran Dangeti <ki...@gmail.com>.
Hi Siddharth,
While running the multi-node we need to take care of the local host of the
slave machine from the error messages the task tracker root directory not
able to get to the masters. Please check and rerun it.
Thanks,
Kiran
On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <si...@gmail.com>wrote:
> Hi,
>
> I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it
> watching this tutorial *
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
> *
>
> When I startup the hadoop, I get the folloing error in *all* the
> tasktrackers.
>
> "
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:*Failed to get system directory
> *...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> "
>
> *But everything looks fine in the webUI. *
>
> When I run a job, I get the following error but the job completes anyways.
> I have* attached the* *screenshots* of the maptask failed error log in
> the UI.
>
> *"*
> 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process
> : 2
> 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
> 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
> attempt_201307091215_0001_m_000001_0, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000001_0:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at
> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at
> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
> 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
> 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0%
> 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16%
> 13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
> attempt_201307091215_0001_m_000000_1, Status : FAILED
> Error initializing attempt_201307091215_0001_m_000000_1:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at
> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
> at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
> at
> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
> at java.lang.Thread.run(Thread.java:724)
>
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
> 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
> outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
> 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16%
> 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/09 12:30:52 INFO mapred.JobClient: Job complete:
> job_201307091215_0001
> 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
> 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1
> 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5
> 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711
> 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51
> 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271
> 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612
> 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51
> 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters
> 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45
> 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized
> bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117
> 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85
> 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage
> (bytes)=602996736
> 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10
> 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=520691712
> 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8
> 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=3025334272
> 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10
>
> *"*
>
>
> Following is the complete tasktracker log on one node after starting the
> hadoop (*not running the job*):
>
> *"*
> 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = dmkd-1/172.31.21.10
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.1.2
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
> 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
> ************************************************************/
> 2013-07-09 12:15:15,722 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-07-09 12:15:15,823 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-07-09 12:15:15,824 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-07-09 12:15:15,824 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> system started
> 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
> Loaded the native-hadoop library
> 2013-07-09 12:15:16,223 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-07-09 12:15:16,230 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
> global filtersafety
> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tasktracker with owner as hduser
> 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
> mapred local directories are: /app/hadoop/tmp/mapred/local
> 2013-07-09 12:15:21,652 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
> registered.
> 2013-07-09 12:15:21,653 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> TaskTrackerMetrics registered.
> 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
> SocketReader
> 2013-07-09 12:15:21,683 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> RpcDetailedActivityForPort36783 registered.
> 2013-07-09 12:15:21,684 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> RpcActivityForPort36783 registered.
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 36783: starting
> 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 36783: starting
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
> TaskTracker up at: 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
> 127.0.0.1/127.0.0.1:36783
> 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
> exited with exit code 0
> 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
> 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> disabled.
> 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache created with max memory = 10485760
> 2013-07-09 12:15:21,785 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> ShuffleServerMetrics registered.
> 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
> 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50060
> 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
> FILE_CACHE_SIZE for mapOutputServlet set to : 2000
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051627_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051327_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0005 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0002 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051500_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051634_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921773
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0003 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051627_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051327_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051135_0005 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307050127_0002 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051500_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051634_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307032034_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051203_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
> Adding job_201307051611_0001 for user-log deletion with
> retainTimeStamp:1373472921775
> 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
> 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed
> to get system directory...
>
> *"*
>
> Please help.
>
>
> Thanks and Regards,
> Siddharth
>
>
>
>