You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ravishankar Nair <ra...@gmail.com> on 2012/05/18 03:49:43 UTC
Why this problem is not solved yet ?
Dear experts,
Today is my tenth day working with Hadoop on installing on my windows
machine. I am trying again and again because , some where someone has
written that it works on Windows with CYGWIN.(And noone has written that
Hadoop wont work on Windows). I am attaching my config files.
Kindly help me, if anything can make this work.A feeble and humble request
to all experts out there.
Here is the error, if you search , you can see thousands have reported this
and there is no solution I got yet, though I tried all ways possible. I am
using Windows XP SP3, Hadoop (tried with five versions so far including
1.0.3). I am running on a single node.(machine WSUSJXLHRN13067 IP:-
192.168.0.16)
When I start Hadoop, no issues in any of the versions
rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3
$ bin/start-all.sh
starting namenode, logging to
/home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-namenode-WSUSJXLHRN13067.out
localhost: starting datanode, logging to
/home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-datanode-WSUSJXLHRN13067.out
localhost: starting secondarynamenode, logging to
/home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-secondarynamenode-WSUSJXLHRN13067.out
starting jobtracker, logging to
/home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-jobtracker-WSUSJXLHRN13067.out
localhost: starting tasktracker, logging to
/home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-tasktracker-WSUSJXLHRN13067.out
When I run the example program, this is what is printed on my console:-
$ bin/hadoop jar hadoop-examples-1.0.3.jar grep input output 'dfs[a-z.]+'
12/05/17 21:44:46 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
12/05/17 21:44:46 WARN snappy.LoadSnappy: Snappy native library not loaded
12/05/17 21:44:46 INFO mapred.FileInputFormat: Total input paths to process
: 16
12/05/17 21:44:47 INFO mapred.JobClient: Running job: job_201205172141_0001
12/05/17 21:44:48 INFO mapred.JobClient: map 0% reduce 0%
Now it is HUNG!!. IN most of the versions this is the behaviour.
Here is the log from JOBTRACKER:-
2012-05-17 21:41:28,037 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.3
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
************************************************************/
2012-05-17 21:41:28,147 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2012-05-17 21:41:28,147 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats registered.
2012-05-17 21:41:28,162 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2012-05-17 21:41:28,162 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics
system started
2012-05-17 21:41:28,209 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
QueueMetrics,q=default registered.
2012-05-17 21:41:28,428 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
registered.
2012-05-17 21:41:28,428 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
exists!
2012-05-17 21:41:28,428 INFO
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Updating the current master key for generating delegation tokens
2012-05-17 21:41:28,428 INFO
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Starting expired delegation token remover thread,
tokenRemoverScanInterval=60 min(s)
2012-05-17 21:41:28,428 INFO
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Updating the current master key for generating delegation tokens
2012-05-17 21:41:28,428 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2012-05-17 21:41:28,428 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2012-05-17 21:41:28,444 INFO org.apache.hadoop.mapred.JobTracker: Starting
jobtracker with owner as rn13067
2012-05-17 21:41:28,475 INFO org.apache.hadoop.ipc.Server: Starting
SocketReader
2012-05-17 21:41:28,475 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
RpcDetailedActivityForPort47111 registered.
2012-05-17 21:41:28,475 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
RpcActivityForPort47111 registered.
2012-05-17 21:41:28,522 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2012-05-17 21:41:28,584 INFO org.apache.hadoop.http.HttpServer: Added
global filtersafety
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-05-17 21:41:28,615 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50030
2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50030
webServer.getConnectors()[0].getLocalPort() returned 50030
2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50030
2012-05-17 21:41:28,615 INFO org.mortbay.log: jetty-6.1.26
2012-05-17 21:41:28,834 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2012-05-17 21:41:28,834 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
registered.
2012-05-17 21:41:28,834 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
JobTrackerMetrics registered.
2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker up at: 47111
2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker webserver: 50030
2012-05-17 21:41:29,225 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2012-05-17 21:41:29,772 INFO org.apache.hadoop.mapred.JobHistory: Creating
DONE folder at file:/C:/cygwin/home/hadoop-1.0.3/logs/history/done
2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker: History
server being initialized in embedded mode
2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobHistoryServer:
Started job history server at: localhost:50030
2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker: Job
History Server web address: localhost:50030
2012-05-17 21:41:29,787 INFO
org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
inactive
2012-05-17 21:41:29,990 INFO org.apache.hadoop.mapred.JobTracker:
Refreshing hosts information
2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
Setting the includes file to
2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
Setting the excludes file to
2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker:
Decommissioning 0 nodes
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 47111: starting
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 47111: starting
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 47111: starting
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 47111: starting
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 47111: starting
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 47111: starting
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 47111: starting
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 6 on 47111: starting
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 7 on 47111: starting
2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker: Starting
RUNNING
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 8 on 47111: starting
2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 9 on 47111: starting
2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
job_201205172141_0001: nMaps=16 nReduces=1 max=-1
2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker: Job
job_201205172141_0001 added successfully for user 'rn13067' to queue
'default'
2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker:
Initializing job_201205172141_0001
2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201205172141_0001
2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.AuditLogger:
USER=rn13067 IP=192.168.0.16 OPERATION=SUBMIT_JOB
TARGET=job_201205172141_0001 RESULT=SUCCESS
*2012-05-17 21:44:47,084 ERROR org.apache.hadoop.mapred.JobHistory: Failed
creating job history log file for job job_201205172141_0001
java.io.IOException: Failed to set permissions of path:
C:\cygwin\home\hadoop-1.0.3\logs\history\job_201205172141_0001_1337305487022_rn13067_grep-search
to 0744*
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:678)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
at
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:286)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:385)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
at
org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:1696)
at org.apache.hadoop.mapred.JobInProgress$3.run(JobInProgress.java:681)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at
org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:678)
at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207)
at
org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Kindly help me. CYGWIN is in the path. (As some one suggested in some
thread).
--
Warmest Regards,
Ravi
Re: Why this problem is not solved yet ?
Posted by Ravi Shankar Nair <ra...@gmail.com>.
Hi Ravi,
Let me try and revert by couple of hours, thanks for input
Sent from my Beethoven
On May 18, 2012, at 3:45 PM, Ravi Prakash <ra...@gmail.com> wrote:
> Hi Ravishankar,
>
> I don't see two very important processes in your jps output. Just like
> there's JobTracker and NameNode, you should also have "TaskTracker" and
> "DataNode". The JobTracker only schedules jobs. To actually run the map
> reduce tasks, it needs TaskTrackers. This is why you see the jobtracker
> accepting your jobs and then getting stuck: because it doesn't have
> TaskTrackers to run that job on.
>
> If I were you, I'd first check why the Datanode is not coming up. All 4
> daemons are necessary for running jobs. The logs for those two should be in
> the same directory in which you find the JT's logs.
>
> Hope this helps.
> Ravi.
>
> On Fri, May 18, 2012 at 5:17 AM, Ravishankar Nair <
> ravishankar.nair@gmail.com> wrote:
>
>> Additionally, attached is the output of the job that I run( I mean the
>> example program named grep)
>>
>>
>> On Fri, May 18, 2012 at 6:15 AM, Ravishankar Nair <
>> ravishankar.nair@gmail.com> wrote:
>>
>>> Hi Ravi,
>>>
>>> Yes , it Running. Here is the output:-
>>> rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3
>>> $ jps
>>> 5068 NameNode
>>> 5836 Jps
>>> 3516 JobTracker
>>>
>>>
>>> Here are the logs from JOBTRACKER:-
>>>
>>> 2012-05-17 21:41:31,772 INFO org.apache.hadoop.mapred.TaskTracker:
>>> STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting TaskTracker
>>>
>>> STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.3
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
>>> ************************************************************/
>>> 2012-05-17 21:41:31,944 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2012-05-17 21:41:31,990 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2012-05-17 21:41:31,990 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2012-05-17 21:41:31,990 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>>> system started
>>> 2012-05-17 21:41:32,256 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2012-05-17 21:41:32,256 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2012-05-17 21:41:32,365 INFO org.mortbay.log: Logging to
>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> org.mortbay.log.Slf4jLog
>>> 2012-05-17 21:41:32,412 INFO org.apache.hadoop.http.HttpServer: Added
>>> global filtersafety
>>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>>> 2012-05-17 21:41:32,428 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>> 2012-05-17 21:41:32,444 INFO org.apache.hadoop.mapred.TaskTracker:
>>> Starting tasktracker with owner as SYSTEM
>>> 2012-05-17 21:41:32,444 INFO org.apache.hadoop.mapred.TaskTracker: Good
>>> mapred local directories are: /tmp/hadoop-SYSTEM/mapred/local
>>> 2012-05-17 21:41:32,459 WARN org.apache.hadoop.util.NativeCodeLoader:
>>> Unable to load native-hadoop library for your platform... using
>>> builtin-java classes where applicable
>>> 2012-05-17 21:41:32,459 ERROR org.apache.hadoop.mapred.TaskTracker: Can
>>> not start task tracker because java.io.IOException: Failed to set
>>> permissions of path: \tmp\hadoop-SYSTEM\mapred\local\ttprivate to 0700
>>> at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>> at
>>> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:728)
>>> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1459)
>>> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
>>>
>>> 2012-05-17 21:41:32,459 INFO org.apache.hadoop.mapred.TaskTracker:
>>> SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down TaskTracker at WSUSJXLHRN13067/192.168.0.16
>>> ************************************************************/
>>>
>>> Any clue? Thanks
>>> Regards,
>>> ravi
>>>
>>>
>>>
>>> On Fri, May 18, 2012 at 12:01 AM, Ravi Prakash <ra...@gmail.com>wrote:
>>>
>>>> Ravishankar,
>>>>
>>>> If you run $ jps, do you see a TaskTracker process running? Can you
>>>> please
>>>> post the tasktracker logs as well?
>>>>
>>>> On Thu, May 17, 2012 at 8:49 PM, Ravishankar Nair <
>>>> ravishankar.nair@gmail.com> wrote:
>>>>
>>>>> Dear experts,
>>>>>
>>>>> Today is my tenth day working with Hadoop on installing on my windows
>>>>> machine. I am trying again and again because , some where someone has
>>>>> written that it works on Windows with CYGWIN.(And noone has written
>>>> that
>>>>> Hadoop wont work on Windows). I am attaching my config files.
>>>>>
>>>>> Kindly help me, if anything can make this work.A feeble and humble
>>>> request
>>>>> to all experts out there.
>>>>>
>>>>> Here is the error, if you search , you can see thousands have reported
>>>>> this and there is no solution I got yet, though I tried all ways
>>>> possible.
>>>>> I am using Windows XP SP3, Hadoop (tried with five versions so far
>>>>> including 1.0.3). I am running on a single node.(machine
>>>> WSUSJXLHRN13067
>>>>> IP:- 192.168.0.16)
>>>>> When I start Hadoop, no issues in any of the versions
>>>>>
>>>>> rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3
>>>>> $ bin/start-all.sh
>>>>> starting namenode, logging to
>>>>>
>>>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-namenode-WSUSJXLHRN13067.out
>>>>> localhost: starting datanode, logging to
>>>>>
>>>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-datanode-WSUSJXLHRN13067.out
>>>>> localhost: starting secondarynamenode, logging to
>>>>>
>>>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-secondarynamenode-WSUSJXLHRN13067.out
>>>>> starting jobtracker, logging to
>>>>>
>>>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-jobtracker-WSUSJXLHRN13067.out
>>>>> localhost: starting tasktracker, logging to
>>>>>
>>>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-tasktracker-WSUSJXLHRN13067.out
>>>>>
>>>>>
>>>>>
>>>>> When I run the example program, this is what is printed on my console:-
>>>>> $ bin/hadoop jar hadoop-examples-1.0.3.jar grep input output
>>>> 'dfs[a-z.]+'
>>>>> 12/05/17 21:44:46 WARN util.NativeCodeLoader: Unable to load
>>>> native-hadoop
>>>>> library for your platform... using builtin-java classes where
>>>> applicable
>>>>> 12/05/17 21:44:46 WARN snappy.LoadSnappy: Snappy native library not
>>>> loaded
>>>>> 12/05/17 21:44:46 INFO mapred.FileInputFormat: Total input paths to
>>>>> process : 16
>>>>> 12/05/17 21:44:47 INFO mapred.JobClient: Running job:
>>>> job_201205172141_0001
>>>>> 12/05/17 21:44:48 INFO mapred.JobClient: map 0% reduce 0%
>>>>>
>>>>>
>>>>> Now it is HUNG!!. IN most of the versions this is the behaviour.
>>>>>
>>>>> Here is the log from JOBTRACKER:-
>>>>>
>>>>>
>>>>> 2012-05-17 21:41:28,037 INFO org.apache.hadoop.mapred.JobTracker:
>>>>> STARTUP_MSG:
>>>>> /************************************************************
>>>>> STARTUP_MSG: Starting JobTracker
>>>>> STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16
>>>>> STARTUP_MSG: args = []
>>>>> STARTUP_MSG: version = 1.0.3
>>>>> STARTUP_MSG: build =
>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>>>> 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
>>>>> ************************************************************/
>>>>> 2012-05-17 21:41:28,147 INFO
>>>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>>>> hadoop-metrics2.properties
>>>>> 2012-05-17 21:41:28,147 INFO
>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>>>> MetricsSystem,sub=Stats registered.
>>>>> 2012-05-17 21:41:28,162 INFO
>>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>>>> period at 10 second(s).
>>>>> 2012-05-17 21:41:28,162 INFO
>>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics
>>>>> system started
>>>>> 2012-05-17 21:41:28,209 INFO
>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>>>> QueueMetrics,q=default registered.
>>>>> 2012-05-17 21:41:28,428 INFO
>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>>> ugi
>>>>> registered.
>>>>> 2012-05-17 21:41:28,428 WARN
>>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
>>>> already
>>>>> exists!
>>>>> 2012-05-17 21:41:28,428 INFO
>>>>>
>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>>>> Updating the current master key for generating delegation tokens
>>>>> 2012-05-17 21:41:28,428 INFO
>>>>>
>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>>>> Starting expired delegation token remover thread,
>>>>> tokenRemoverScanInterval=60 min(s)
>>>>> 2012-05-17 21:41:28,428 INFO
>>>>>
>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>>>> Updating the current master key for generating delegation tokens
>>>>> 2012-05-17 21:41:28,428 INFO org.apache.hadoop.mapred.JobTracker:
>>>>> Scheduler configured with (memSizeForMapSlotOnJT,
>>>> memSizeForReduceSlotOnJT,
>>>>> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
>>>>> 2012-05-17 21:41:28,428 INFO org.apache.hadoop.util.HostsFileReader:
>>>>> Refreshing hosts (include/exclude) list
>>>>> 2012-05-17 21:41:28,444 INFO org.apache.hadoop.mapred.JobTracker:
>>>> Starting
>>>>> jobtracker with owner as rn13067
>>>>> 2012-05-17 21:41:28,475 INFO org.apache.hadoop.ipc.Server: Starting
>>>>> SocketReader
>>>>> 2012-05-17 21:41:28,475 INFO
>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>>>> RpcDetailedActivityForPort47111 registered.
>>>>> 2012-05-17 21:41:28,475 INFO
>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>>>> RpcActivityForPort47111 registered.
>>>>> 2012-05-17 21:41:28,522 INFO org.mortbay.log: Logging to
>>>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>>>> org.mortbay.log.Slf4jLog
>>>>> 2012-05-17 21:41:28,584 INFO org.apache.hadoop.http.HttpServer: Added
>>>>> global filtersafety
>>>>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>>>>> 2012-05-17 21:41:28,615 WARN org.apache.hadoop.util.NativeCodeLoader:
>>>>> Unable to load native-hadoop library for your platform... using
>>>>> builtin-java classes where applicable
>>>>> 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Port
>>>>> returned by webServer.getConnectors()[0].getLocalPort() before open()
>>>> is
>>>>> -1. Opening the listener on 50030
>>>>> 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer:
>>>>> listener.getLocalPort() returned 50030
>>>>> webServer.getConnectors()[0].getLocalPort() returned 50030
>>>>> 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Jetty
>>>>> bound to port 50030
>>>>> 2012-05-17 21:41:28,615 INFO org.mortbay.log: jetty-6.1.26
>>>>> 2012-05-17 21:41:28,834 INFO org.mortbay.log: Started
>>>>> SelectChannelConnector@0.0.0.0:50030
>>>>> 2012-05-17 21:41:28,834 INFO
>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>>> jvm
>>>>> registered.
>>>>> 2012-05-17 21:41:28,834 INFO
>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>>>> JobTrackerMetrics registered.
>>>>> 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
>>>>> JobTracker up at: 47111
>>>>> 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
>>>>> JobTracker webserver: 50030
>>>>> 2012-05-17 21:41:29,225 INFO org.apache.hadoop.mapred.JobTracker:
>>>> Cleaning
>>>>> up the system directory
>>>>> 2012-05-17 21:41:29,772 INFO org.apache.hadoop.mapred.JobHistory:
>>>> Creating
>>>>> DONE folder at file:/C:/cygwin/home/hadoop-1.0.3/logs/history/done
>>>>> 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker:
>>>> History
>>>>> server being initialized in embedded mode
>>>>> 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobHistoryServer:
>>>>> Started job history server at: localhost:50030
>>>>> 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker: Job
>>>>> History Server web address: localhost:50030
>>>>> 2012-05-17 21:41:29,787 INFO
>>>>> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store
>>>> is
>>>>> inactive
>>>>> 2012-05-17 21:41:29,990 INFO org.apache.hadoop.mapred.JobTracker:
>>>>> Refreshing hosts information
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
>>>>> Setting the includes file to
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
>>>>> Setting the excludes file to
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
>>>>> Refreshing hosts (include/exclude) list
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker:
>>>>> Decommissioning 0 nodes
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> Responder: starting
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> listener on 47111: starting
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 0 on 47111: starting
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 1 on 47111: starting
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 2 on 47111: starting
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 3 on 47111: starting
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 4 on 47111: starting
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 5 on 47111: starting
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 6 on 47111: starting
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 7 on 47111: starting
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker:
>>>> Starting
>>>>> RUNNING
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 8 on 47111: starting
>>>>> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 9 on 47111: starting
>>>>> 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
>>>>> job_201205172141_0001: nMaps=16 nReduces=1 max=-1
>>>>> 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker: Job
>>>>> job_201205172141_0001 added successfully for user 'rn13067' to queue
>>>>> 'default'
>>>>> 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker:
>>>>> Initializing job_201205172141_0001
>>>>> 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
>>>>> Initializing job_201205172141_0001
>>>>> 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.AuditLogger:
>>>>> USER=rn13067 IP=192.168.0.16 OPERATION=SUBMIT_JOB
>>>>> TARGET=job_201205172141_0001 RESULT=SUCCESS
>>>>> *2012-05-17 21:44:47,084 ERROR org.apache.hadoop.mapred.JobHistory:
>>>>> Failed creating job history log file for job job_201205172141_0001
>>>>> java.io.IOException: Failed to set permissions of path:
>>>>>
>>>> C:\cygwin\home\hadoop-1.0.3\logs\history\job_201205172141_0001_1337305487022_rn13067_grep-search
>>>>> to 0744*
>>>>> at
>>>> org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
>>>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:678)
>>>>> at
>>>>>
>>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>>>> at
>>>>>
>>>> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:286)
>>>>> at
>>>>>
>>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:385)
>>>>> at
>>>>>
>>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
>>>>> at
>>>>>
>>>> org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:1696)
>>>>> at
>>>> org.apache.hadoop.mapred.JobInProgress$3.run(JobInProgress.java:681)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>> at
>>>>>
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>>>> at
>>>>>
>>>> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:678)
>>>>> at
>>>> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207)
>>>>> at
>>>>>
>>>> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
>>>>> at
>>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> Kindly help me. CYGWIN is in the path. (As some one suggested in some
>>>>> thread).
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Warmest Regards,
>>>>>
>>>>> Ravi
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Warmest Regards,
>>>
>>> Ravi Shankar
>>>
>>
>>
>>
>> --
>> Warmest Regards,
>>
>> Ravi Shankar
>>
Re: Why this problem is not solved yet ?
Posted by Ravi Prakash <ra...@gmail.com>.
Hi Ravishankar,
I don't see two very important processes in your jps output. Just like
there's JobTracker and NameNode, you should also have "TaskTracker" and
"DataNode". The JobTracker only schedules jobs. To actually run the map
reduce tasks, it needs TaskTrackers. This is why you see the jobtracker
accepting your jobs and then getting stuck: because it doesn't have
TaskTrackers to run that job on.
If I were you, I'd first check why the Datanode is not coming up. All 4
daemons are necessary for running jobs. The logs for those two should be in
the same directory in which you find the JT's logs.
Hope this helps.
Ravi.
On Fri, May 18, 2012 at 5:17 AM, Ravishankar Nair <
ravishankar.nair@gmail.com> wrote:
> Additionally, attached is the output of the job that I run( I mean the
> example program named grep)
>
>
> On Fri, May 18, 2012 at 6:15 AM, Ravishankar Nair <
> ravishankar.nair@gmail.com> wrote:
>
>> Hi Ravi,
>>
>> Yes , it Running. Here is the output:-
>> rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3
>> $ jps
>> 5068 NameNode
>> 5836 Jps
>> 3516 JobTracker
>>
>>
>> Here are the logs from JOBTRACKER:-
>>
>> 2012-05-17 21:41:31,772 INFO org.apache.hadoop.mapred.TaskTracker:
>> STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting TaskTracker
>>
>> STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.0.3
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>> 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
>> ************************************************************/
>> 2012-05-17 21:41:31,944 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2012-05-17 21:41:31,990 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2012-05-17 21:41:31,990 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2012-05-17 21:41:31,990 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>> system started
>> 2012-05-17 21:41:32,256 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>> registered.
>> 2012-05-17 21:41:32,256 WARN
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>> exists!
>> 2012-05-17 21:41:32,365 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2012-05-17 21:41:32,412 INFO org.apache.hadoop.http.HttpServer: Added
>> global filtersafety
>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2012-05-17 21:41:32,428 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>> 2012-05-17 21:41:32,444 INFO org.apache.hadoop.mapred.TaskTracker:
>> Starting tasktracker with owner as SYSTEM
>> 2012-05-17 21:41:32,444 INFO org.apache.hadoop.mapred.TaskTracker: Good
>> mapred local directories are: /tmp/hadoop-SYSTEM/mapred/local
>> 2012-05-17 21:41:32,459 WARN org.apache.hadoop.util.NativeCodeLoader:
>> Unable to load native-hadoop library for your platform... using
>> builtin-java classes where applicable
>> 2012-05-17 21:41:32,459 ERROR org.apache.hadoop.mapred.TaskTracker: Can
>> not start task tracker because java.io.IOException: Failed to set
>> permissions of path: \tmp\hadoop-SYSTEM\mapred\local\ttprivate to 0700
>> at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>> at
>> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
>> at
>> org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:728)
>> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1459)
>> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
>>
>> 2012-05-17 21:41:32,459 INFO org.apache.hadoop.mapred.TaskTracker:
>> SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down TaskTracker at WSUSJXLHRN13067/192.168.0.16
>> ************************************************************/
>>
>> Any clue? Thanks
>> Regards,
>> ravi
>>
>>
>>
>> On Fri, May 18, 2012 at 12:01 AM, Ravi Prakash <ra...@gmail.com>wrote:
>>
>>> Ravishankar,
>>>
>>> If you run $ jps, do you see a TaskTracker process running? Can you
>>> please
>>> post the tasktracker logs as well?
>>>
>>> On Thu, May 17, 2012 at 8:49 PM, Ravishankar Nair <
>>> ravishankar.nair@gmail.com> wrote:
>>>
>>> > Dear experts,
>>> >
>>> > Today is my tenth day working with Hadoop on installing on my windows
>>> > machine. I am trying again and again because , some where someone has
>>> > written that it works on Windows with CYGWIN.(And noone has written
>>> that
>>> > Hadoop wont work on Windows). I am attaching my config files.
>>> >
>>> > Kindly help me, if anything can make this work.A feeble and humble
>>> request
>>> > to all experts out there.
>>> >
>>> > Here is the error, if you search , you can see thousands have reported
>>> > this and there is no solution I got yet, though I tried all ways
>>> possible.
>>> > I am using Windows XP SP3, Hadoop (tried with five versions so far
>>> > including 1.0.3). I am running on a single node.(machine
>>> WSUSJXLHRN13067
>>> > IP:- 192.168.0.16)
>>> > When I start Hadoop, no issues in any of the versions
>>> >
>>> > rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3
>>> > $ bin/start-all.sh
>>> > starting namenode, logging to
>>> >
>>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-namenode-WSUSJXLHRN13067.out
>>> > localhost: starting datanode, logging to
>>> >
>>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-datanode-WSUSJXLHRN13067.out
>>> > localhost: starting secondarynamenode, logging to
>>> >
>>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-secondarynamenode-WSUSJXLHRN13067.out
>>> > starting jobtracker, logging to
>>> >
>>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-jobtracker-WSUSJXLHRN13067.out
>>> > localhost: starting tasktracker, logging to
>>> >
>>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-tasktracker-WSUSJXLHRN13067.out
>>> >
>>> >
>>> >
>>> > When I run the example program, this is what is printed on my console:-
>>> > $ bin/hadoop jar hadoop-examples-1.0.3.jar grep input output
>>> 'dfs[a-z.]+'
>>> > 12/05/17 21:44:46 WARN util.NativeCodeLoader: Unable to load
>>> native-hadoop
>>> > library for your platform... using builtin-java classes where
>>> applicable
>>> > 12/05/17 21:44:46 WARN snappy.LoadSnappy: Snappy native library not
>>> loaded
>>> > 12/05/17 21:44:46 INFO mapred.FileInputFormat: Total input paths to
>>> > process : 16
>>> > 12/05/17 21:44:47 INFO mapred.JobClient: Running job:
>>> job_201205172141_0001
>>> > 12/05/17 21:44:48 INFO mapred.JobClient: map 0% reduce 0%
>>> >
>>> >
>>> > Now it is HUNG!!. IN most of the versions this is the behaviour.
>>> >
>>> > Here is the log from JOBTRACKER:-
>>> >
>>> >
>>> > 2012-05-17 21:41:28,037 INFO org.apache.hadoop.mapred.JobTracker:
>>> > STARTUP_MSG:
>>> > /************************************************************
>>> > STARTUP_MSG: Starting JobTracker
>>> > STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16
>>> > STARTUP_MSG: args = []
>>> > STARTUP_MSG: version = 1.0.3
>>> > STARTUP_MSG: build =
>>> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> > 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
>>> > ************************************************************/
>>> > 2012-05-17 21:41:28,147 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> > hadoop-metrics2.properties
>>> > 2012-05-17 21:41:28,147 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> > MetricsSystem,sub=Stats registered.
>>> > 2012-05-17 21:41:28,162 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> > period at 10 second(s).
>>> > 2012-05-17 21:41:28,162 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics
>>> > system started
>>> > 2012-05-17 21:41:28,209 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> > QueueMetrics,q=default registered.
>>> > 2012-05-17 21:41:28,428 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> ugi
>>> > registered.
>>> > 2012-05-17 21:41:28,428 WARN
>>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
>>> already
>>> > exists!
>>> > 2012-05-17 21:41:28,428 INFO
>>> >
>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>> > Updating the current master key for generating delegation tokens
>>> > 2012-05-17 21:41:28,428 INFO
>>> >
>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>> > Starting expired delegation token remover thread,
>>> > tokenRemoverScanInterval=60 min(s)
>>> > 2012-05-17 21:41:28,428 INFO
>>> >
>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>> > Updating the current master key for generating delegation tokens
>>> > 2012-05-17 21:41:28,428 INFO org.apache.hadoop.mapred.JobTracker:
>>> > Scheduler configured with (memSizeForMapSlotOnJT,
>>> memSizeForReduceSlotOnJT,
>>> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
>>> > 2012-05-17 21:41:28,428 INFO org.apache.hadoop.util.HostsFileReader:
>>> > Refreshing hosts (include/exclude) list
>>> > 2012-05-17 21:41:28,444 INFO org.apache.hadoop.mapred.JobTracker:
>>> Starting
>>> > jobtracker with owner as rn13067
>>> > 2012-05-17 21:41:28,475 INFO org.apache.hadoop.ipc.Server: Starting
>>> > SocketReader
>>> > 2012-05-17 21:41:28,475 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> > RpcDetailedActivityForPort47111 registered.
>>> > 2012-05-17 21:41:28,475 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> > RpcActivityForPort47111 registered.
>>> > 2012-05-17 21:41:28,522 INFO org.mortbay.log: Logging to
>>> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> > org.mortbay.log.Slf4jLog
>>> > 2012-05-17 21:41:28,584 INFO org.apache.hadoop.http.HttpServer: Added
>>> > global filtersafety
>>> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>>> > 2012-05-17 21:41:28,615 WARN org.apache.hadoop.util.NativeCodeLoader:
>>> > Unable to load native-hadoop library for your platform... using
>>> > builtin-java classes where applicable
>>> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Port
>>> > returned by webServer.getConnectors()[0].getLocalPort() before open()
>>> is
>>> > -1. Opening the listener on 50030
>>> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer:
>>> > listener.getLocalPort() returned 50030
>>> > webServer.getConnectors()[0].getLocalPort() returned 50030
>>> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Jetty
>>> > bound to port 50030
>>> > 2012-05-17 21:41:28,615 INFO org.mortbay.log: jetty-6.1.26
>>> > 2012-05-17 21:41:28,834 INFO org.mortbay.log: Started
>>> > SelectChannelConnector@0.0.0.0:50030
>>> > 2012-05-17 21:41:28,834 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> jvm
>>> > registered.
>>> > 2012-05-17 21:41:28,834 INFO
>>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> > JobTrackerMetrics registered.
>>> > 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
>>> > JobTracker up at: 47111
>>> > 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
>>> > JobTracker webserver: 50030
>>> > 2012-05-17 21:41:29,225 INFO org.apache.hadoop.mapred.JobTracker:
>>> Cleaning
>>> > up the system directory
>>> > 2012-05-17 21:41:29,772 INFO org.apache.hadoop.mapred.JobHistory:
>>> Creating
>>> > DONE folder at file:/C:/cygwin/home/hadoop-1.0.3/logs/history/done
>>> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker:
>>> History
>>> > server being initialized in embedded mode
>>> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobHistoryServer:
>>> > Started job history server at: localhost:50030
>>> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker: Job
>>> > History Server web address: localhost:50030
>>> > 2012-05-17 21:41:29,787 INFO
>>> > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store
>>> is
>>> > inactive
>>> > 2012-05-17 21:41:29,990 INFO org.apache.hadoop.mapred.JobTracker:
>>> > Refreshing hosts information
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
>>> > Setting the includes file to
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
>>> > Setting the excludes file to
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
>>> > Refreshing hosts (include/exclude) list
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker:
>>> > Decommissioning 0 nodes
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > Responder: starting
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > listener on 47111: starting
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > handler 0 on 47111: starting
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > handler 1 on 47111: starting
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > handler 2 on 47111: starting
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > handler 3 on 47111: starting
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > handler 4 on 47111: starting
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > handler 5 on 47111: starting
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > handler 6 on 47111: starting
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > handler 7 on 47111: starting
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker:
>>> Starting
>>> > RUNNING
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > handler 8 on 47111: starting
>>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > handler 9 on 47111: starting
>>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
>>> > job_201205172141_0001: nMaps=16 nReduces=1 max=-1
>>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker: Job
>>> > job_201205172141_0001 added successfully for user 'rn13067' to queue
>>> > 'default'
>>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker:
>>> > Initializing job_201205172141_0001
>>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
>>> > Initializing job_201205172141_0001
>>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.AuditLogger:
>>> > USER=rn13067 IP=192.168.0.16 OPERATION=SUBMIT_JOB
>>> > TARGET=job_201205172141_0001 RESULT=SUCCESS
>>> > *2012-05-17 21:44:47,084 ERROR org.apache.hadoop.mapred.JobHistory:
>>> > Failed creating job history log file for job job_201205172141_0001
>>> > java.io.IOException: Failed to set permissions of path:
>>> >
>>> C:\cygwin\home\hadoop-1.0.3\logs\history\job_201205172141_0001_1337305487022_rn13067_grep-search
>>> > to 0744*
>>> > at
>>> org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
>>> > at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:678)
>>> > at
>>> >
>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>> > at
>>> >
>>> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:286)
>>> > at
>>> >
>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:385)
>>> > at
>>> >
>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
>>> > at
>>> >
>>> org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:1696)
>>> > at
>>> org.apache.hadoop.mapred.JobInProgress$3.run(JobInProgress.java:681)
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>>> > at
>>> >
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>> > at
>>> >
>>> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:678)
>>> > at
>>> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207)
>>> > at
>>> >
>>> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
>>> > at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> > at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> > at java.lang.Thread.run(Thread.java:662)
>>> >
>>> > Kindly help me. CYGWIN is in the path. (As some one suggested in some
>>> > thread).
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Warmest Regards,
>>> >
>>> > Ravi
>>> >
>>>
>>
>>
>>
>> --
>> Warmest Regards,
>>
>> Ravi Shankar
>>
>
>
>
> --
> Warmest Regards,
>
> Ravi Shankar
>
Re: Why this problem is not solved yet ?
Posted by Ravishankar Nair <ra...@gmail.com>.
Additionally, attached is the output of the job that I run( I mean the
example program named grep)
On Fri, May 18, 2012 at 6:15 AM, Ravishankar Nair <
ravishankar.nair@gmail.com> wrote:
> Hi Ravi,
>
> Yes , it Running. Here is the output:-
> rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3
> $ jps
> 5068 NameNode
> 5836 Jps
> 3516 JobTracker
>
>
> Here are the logs from JOBTRACKER:-
>
> 2012-05-17 21:41:31,772 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
>
> STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.3
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
> ************************************************************/
> 2012-05-17 21:41:31,944 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2012-05-17 21:41:31,990 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2012-05-17 21:41:31,990 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2012-05-17 21:41:31,990 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> system started
> 2012-05-17 21:41:32,256 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2012-05-17 21:41:32,256 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2012-05-17 21:41:32,365 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2012-05-17 21:41:32,412 INFO org.apache.hadoop.http.HttpServer: Added
> global filtersafety
> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2012-05-17 21:41:32,428 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2012-05-17 21:41:32,444 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tasktracker with owner as SYSTEM
> 2012-05-17 21:41:32,444 INFO org.apache.hadoop.mapred.TaskTracker: Good
> mapred local directories are: /tmp/hadoop-SYSTEM/mapred/local
> 2012-05-17 21:41:32,459 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2012-05-17 21:41:32,459 ERROR org.apache.hadoop.mapred.TaskTracker: Can
> not start task tracker because java.io.IOException: Failed to set
> permissions of path: \tmp\hadoop-SYSTEM\mapred\local\ttprivate to 0700
> at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at
> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
> at
> org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:728)
> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1459)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
>
> 2012-05-17 21:41:32,459 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker at WSUSJXLHRN13067/192.168.0.16
> ************************************************************/
>
> Any clue? Thanks
> Regards,
> ravi
>
>
>
> On Fri, May 18, 2012 at 12:01 AM, Ravi Prakash <ra...@gmail.com>wrote:
>
>> Ravishankar,
>>
>> If you run $ jps, do you see a TaskTracker process running? Can you please
>> post the tasktracker logs as well?
>>
>> On Thu, May 17, 2012 at 8:49 PM, Ravishankar Nair <
>> ravishankar.nair@gmail.com> wrote:
>>
>> > Dear experts,
>> >
>> > Today is my tenth day working with Hadoop on installing on my windows
>> > machine. I am trying again and again because , some where someone has
>> > written that it works on Windows with CYGWIN.(And noone has written that
>> > Hadoop wont work on Windows). I am attaching my config files.
>> >
>> > Kindly help me, if anything can make this work.A feeble and humble
>> request
>> > to all experts out there.
>> >
>> > Here is the error, if you search , you can see thousands have reported
>> > this and there is no solution I got yet, though I tried all ways
>> possible.
>> > I am using Windows XP SP3, Hadoop (tried with five versions so far
>> > including 1.0.3). I am running on a single node.(machine
>> WSUSJXLHRN13067
>> > IP:- 192.168.0.16)
>> > When I start Hadoop, no issues in any of the versions
>> >
>> > rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3
>> > $ bin/start-all.sh
>> > starting namenode, logging to
>> >
>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-namenode-WSUSJXLHRN13067.out
>> > localhost: starting datanode, logging to
>> >
>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-datanode-WSUSJXLHRN13067.out
>> > localhost: starting secondarynamenode, logging to
>> >
>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-secondarynamenode-WSUSJXLHRN13067.out
>> > starting jobtracker, logging to
>> >
>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-jobtracker-WSUSJXLHRN13067.out
>> > localhost: starting tasktracker, logging to
>> >
>> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-tasktracker-WSUSJXLHRN13067.out
>> >
>> >
>> >
>> > When I run the example program, this is what is printed on my console:-
>> > $ bin/hadoop jar hadoop-examples-1.0.3.jar grep input output
>> 'dfs[a-z.]+'
>> > 12/05/17 21:44:46 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop
>> > library for your platform... using builtin-java classes where applicable
>> > 12/05/17 21:44:46 WARN snappy.LoadSnappy: Snappy native library not
>> loaded
>> > 12/05/17 21:44:46 INFO mapred.FileInputFormat: Total input paths to
>> > process : 16
>> > 12/05/17 21:44:47 INFO mapred.JobClient: Running job:
>> job_201205172141_0001
>> > 12/05/17 21:44:48 INFO mapred.JobClient: map 0% reduce 0%
>> >
>> >
>> > Now it is HUNG!!. IN most of the versions this is the behaviour.
>> >
>> > Here is the log from JOBTRACKER:-
>> >
>> >
>> > 2012-05-17 21:41:28,037 INFO org.apache.hadoop.mapred.JobTracker:
>> > STARTUP_MSG:
>> > /************************************************************
>> > STARTUP_MSG: Starting JobTracker
>> > STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16
>> > STARTUP_MSG: args = []
>> > STARTUP_MSG: version = 1.0.3
>> > STARTUP_MSG: build =
>> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>> > 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
>> > ************************************************************/
>> > 2012-05-17 21:41:28,147 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> > hadoop-metrics2.properties
>> > 2012-05-17 21:41:28,147 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> > MetricsSystem,sub=Stats registered.
>> > 2012-05-17 21:41:28,162 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> > period at 10 second(s).
>> > 2012-05-17 21:41:28,162 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics
>> > system started
>> > 2012-05-17 21:41:28,209 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> > QueueMetrics,q=default registered.
>> > 2012-05-17 21:41:28,428 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> ugi
>> > registered.
>> > 2012-05-17 21:41:28,428 WARN
>> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
>> already
>> > exists!
>> > 2012-05-17 21:41:28,428 INFO
>> >
>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>> > Updating the current master key for generating delegation tokens
>> > 2012-05-17 21:41:28,428 INFO
>> >
>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>> > Starting expired delegation token remover thread,
>> > tokenRemoverScanInterval=60 min(s)
>> > 2012-05-17 21:41:28,428 INFO
>> >
>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>> > Updating the current master key for generating delegation tokens
>> > 2012-05-17 21:41:28,428 INFO org.apache.hadoop.mapred.JobTracker:
>> > Scheduler configured with (memSizeForMapSlotOnJT,
>> memSizeForReduceSlotOnJT,
>> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
>> > 2012-05-17 21:41:28,428 INFO org.apache.hadoop.util.HostsFileReader:
>> > Refreshing hosts (include/exclude) list
>> > 2012-05-17 21:41:28,444 INFO org.apache.hadoop.mapred.JobTracker:
>> Starting
>> > jobtracker with owner as rn13067
>> > 2012-05-17 21:41:28,475 INFO org.apache.hadoop.ipc.Server: Starting
>> > SocketReader
>> > 2012-05-17 21:41:28,475 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> > RpcDetailedActivityForPort47111 registered.
>> > 2012-05-17 21:41:28,475 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> > RpcActivityForPort47111 registered.
>> > 2012-05-17 21:41:28,522 INFO org.mortbay.log: Logging to
>> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> > org.mortbay.log.Slf4jLog
>> > 2012-05-17 21:41:28,584 INFO org.apache.hadoop.http.HttpServer: Added
>> > global filtersafety
>> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> > 2012-05-17 21:41:28,615 WARN org.apache.hadoop.util.NativeCodeLoader:
>> > Unable to load native-hadoop library for your platform... using
>> > builtin-java classes where applicable
>> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Port
>> > returned by webServer.getConnectors()[0].getLocalPort() before open() is
>> > -1. Opening the listener on 50030
>> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer:
>> > listener.getLocalPort() returned 50030
>> > webServer.getConnectors()[0].getLocalPort() returned 50030
>> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Jetty
>> > bound to port 50030
>> > 2012-05-17 21:41:28,615 INFO org.mortbay.log: jetty-6.1.26
>> > 2012-05-17 21:41:28,834 INFO org.mortbay.log: Started
>> > SelectChannelConnector@0.0.0.0:50030
>> > 2012-05-17 21:41:28,834 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> jvm
>> > registered.
>> > 2012-05-17 21:41:28,834 INFO
>> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> > JobTrackerMetrics registered.
>> > 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
>> > JobTracker up at: 47111
>> > 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
>> > JobTracker webserver: 50030
>> > 2012-05-17 21:41:29,225 INFO org.apache.hadoop.mapred.JobTracker:
>> Cleaning
>> > up the system directory
>> > 2012-05-17 21:41:29,772 INFO org.apache.hadoop.mapred.JobHistory:
>> Creating
>> > DONE folder at file:/C:/cygwin/home/hadoop-1.0.3/logs/history/done
>> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker:
>> History
>> > server being initialized in embedded mode
>> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobHistoryServer:
>> > Started job history server at: localhost:50030
>> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker: Job
>> > History Server web address: localhost:50030
>> > 2012-05-17 21:41:29,787 INFO
>> > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
>> > inactive
>> > 2012-05-17 21:41:29,990 INFO org.apache.hadoop.mapred.JobTracker:
>> > Refreshing hosts information
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
>> > Setting the includes file to
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
>> > Setting the excludes file to
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
>> > Refreshing hosts (include/exclude) list
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker:
>> > Decommissioning 0 nodes
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > Responder: starting
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > listener on 47111: starting
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 0 on 47111: starting
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 1 on 47111: starting
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 2 on 47111: starting
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 3 on 47111: starting
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 4 on 47111: starting
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 5 on 47111: starting
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 6 on 47111: starting
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 7 on 47111: starting
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker:
>> Starting
>> > RUNNING
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 8 on 47111: starting
>> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 9 on 47111: starting
>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
>> > job_201205172141_0001: nMaps=16 nReduces=1 max=-1
>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker: Job
>> > job_201205172141_0001 added successfully for user 'rn13067' to queue
>> > 'default'
>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker:
>> > Initializing job_201205172141_0001
>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
>> > Initializing job_201205172141_0001
>> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.AuditLogger:
>> > USER=rn13067 IP=192.168.0.16 OPERATION=SUBMIT_JOB
>> > TARGET=job_201205172141_0001 RESULT=SUCCESS
>> > *2012-05-17 21:44:47,084 ERROR org.apache.hadoop.mapred.JobHistory:
>> > Failed creating job history log file for job job_201205172141_0001
>> > java.io.IOException: Failed to set permissions of path:
>> >
>> C:\cygwin\home\hadoop-1.0.3\logs\history\job_201205172141_0001_1337305487022_rn13067_grep-search
>> > to 0744*
>> > at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
>> > at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:678)
>> > at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>> > at
>> >
>> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:286)
>> > at
>> >
>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:385)
>> > at
>> >
>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
>> > at
>> >
>> org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:1696)
>> > at
>> org.apache.hadoop.mapred.JobInProgress$3.run(JobInProgress.java:681)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>> > at
>> > org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:678)
>> > at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207)
>> > at
>> >
>> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > at java.lang.Thread.run(Thread.java:662)
>> >
>> > Kindly help me. CYGWIN is in the path. (As some one suggested in some
>> > thread).
>> >
>> >
>> >
>> >
>> > --
>> > Warmest Regards,
>> >
>> > Ravi
>> >
>>
>
>
>
> --
> Warmest Regards,
>
> Ravi Shankar
>
--
Warmest Regards,
Ravi Shankar
Re: Why this problem is not solved yet ?
Posted by Ravishankar Nair <ra...@gmail.com>.
Hi Ravi,
Yes , it Running. Here is the output:-
rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3
$ jps
5068 NameNode
5836 Jps
3516 JobTracker
Here are the logs from JOBTRACKER:-
2012-05-17 21:41:31,772 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.3
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
************************************************************/
2012-05-17 21:41:31,944 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2012-05-17 21:41:31,990 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats registered.
2012-05-17 21:41:31,990 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2012-05-17 21:41:31,990 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
system started
2012-05-17 21:41:32,256 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
registered.
2012-05-17 21:41:32,256 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
exists!
2012-05-17 21:41:32,365 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2012-05-17 21:41:32,412 INFO org.apache.hadoop.http.HttpServer: Added
global filtersafety
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-05-17 21:41:32,428 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2012-05-17 21:41:32,444 INFO org.apache.hadoop.mapred.TaskTracker: Starting
tasktracker with owner as SYSTEM
2012-05-17 21:41:32,444 INFO org.apache.hadoop.mapred.TaskTracker: Good
mapred local directories are: /tmp/hadoop-SYSTEM/mapred/local
2012-05-17 21:41:32,459 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2012-05-17 21:41:32,459 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
start task tracker because java.io.IOException: Failed to set permissions
of path: \tmp\hadoop-SYSTEM\mapred\local\ttprivate to 0700
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
at
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:728)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1459)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
2012-05-17 21:41:32,459 INFO org.apache.hadoop.mapred.TaskTracker:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at WSUSJXLHRN13067/192.168.0.16
************************************************************/
Any clue? Thanks
Regards,
ravi
On Fri, May 18, 2012 at 12:01 AM, Ravi Prakash <ra...@gmail.com> wrote:
> Ravishankar,
>
> If you run $ jps, do you see a TaskTracker process running? Can you please
> post the tasktracker logs as well?
>
> On Thu, May 17, 2012 at 8:49 PM, Ravishankar Nair <
> ravishankar.nair@gmail.com> wrote:
>
> > Dear experts,
> >
> > Today is my tenth day working with Hadoop on installing on my windows
> > machine. I am trying again and again because , some where someone has
> > written that it works on Windows with CYGWIN.(And noone has written that
> > Hadoop wont work on Windows). I am attaching my config files.
> >
> > Kindly help me, if anything can make this work.A feeble and humble
> request
> > to all experts out there.
> >
> > Here is the error, if you search , you can see thousands have reported
> > this and there is no solution I got yet, though I tried all ways
> possible.
> > I am using Windows XP SP3, Hadoop (tried with five versions so far
> > including 1.0.3). I am running on a single node.(machine WSUSJXLHRN13067
> > IP:- 192.168.0.16)
> > When I start Hadoop, no issues in any of the versions
> >
> > rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3
> > $ bin/start-all.sh
> > starting namenode, logging to
> >
> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-namenode-WSUSJXLHRN13067.out
> > localhost: starting datanode, logging to
> >
> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-datanode-WSUSJXLHRN13067.out
> > localhost: starting secondarynamenode, logging to
> >
> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-secondarynamenode-WSUSJXLHRN13067.out
> > starting jobtracker, logging to
> >
> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-jobtracker-WSUSJXLHRN13067.out
> > localhost: starting tasktracker, logging to
> >
> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-tasktracker-WSUSJXLHRN13067.out
> >
> >
> >
> > When I run the example program, this is what is printed on my console:-
> > $ bin/hadoop jar hadoop-examples-1.0.3.jar grep input output
> 'dfs[a-z.]+'
> > 12/05/17 21:44:46 WARN util.NativeCodeLoader: Unable to load
> native-hadoop
> > library for your platform... using builtin-java classes where applicable
> > 12/05/17 21:44:46 WARN snappy.LoadSnappy: Snappy native library not
> loaded
> > 12/05/17 21:44:46 INFO mapred.FileInputFormat: Total input paths to
> > process : 16
> > 12/05/17 21:44:47 INFO mapred.JobClient: Running job:
> job_201205172141_0001
> > 12/05/17 21:44:48 INFO mapred.JobClient: map 0% reduce 0%
> >
> >
> > Now it is HUNG!!. IN most of the versions this is the behaviour.
> >
> > Here is the log from JOBTRACKER:-
> >
> >
> > 2012-05-17 21:41:28,037 INFO org.apache.hadoop.mapred.JobTracker:
> > STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting JobTracker
> > STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16
> > STARTUP_MSG: args = []
> > STARTUP_MSG: version = 1.0.3
> > STARTUP_MSG: build =
> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> > 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
> > ************************************************************/
> > 2012-05-17 21:41:28,147 INFO
> > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> > hadoop-metrics2.properties
> > 2012-05-17 21:41:28,147 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > MetricsSystem,sub=Stats registered.
> > 2012-05-17 21:41:28,162 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> > period at 10 second(s).
> > 2012-05-17 21:41:28,162 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics
> > system started
> > 2012-05-17 21:41:28,209 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > QueueMetrics,q=default registered.
> > 2012-05-17 21:41:28,428 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> ugi
> > registered.
> > 2012-05-17 21:41:28,428 WARN
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
> already
> > exists!
> > 2012-05-17 21:41:28,428 INFO
> >
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> > Updating the current master key for generating delegation tokens
> > 2012-05-17 21:41:28,428 INFO
> >
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> > Starting expired delegation token remover thread,
> > tokenRemoverScanInterval=60 min(s)
> > 2012-05-17 21:41:28,428 INFO
> >
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> > Updating the current master key for generating delegation tokens
> > 2012-05-17 21:41:28,428 INFO org.apache.hadoop.mapred.JobTracker:
> > Scheduler configured with (memSizeForMapSlotOnJT,
> memSizeForReduceSlotOnJT,
> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> > 2012-05-17 21:41:28,428 INFO org.apache.hadoop.util.HostsFileReader:
> > Refreshing hosts (include/exclude) list
> > 2012-05-17 21:41:28,444 INFO org.apache.hadoop.mapred.JobTracker:
> Starting
> > jobtracker with owner as rn13067
> > 2012-05-17 21:41:28,475 INFO org.apache.hadoop.ipc.Server: Starting
> > SocketReader
> > 2012-05-17 21:41:28,475 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > RpcDetailedActivityForPort47111 registered.
> > 2012-05-17 21:41:28,475 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > RpcActivityForPort47111 registered.
> > 2012-05-17 21:41:28,522 INFO org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2012-05-17 21:41:28,584 INFO org.apache.hadoop.http.HttpServer: Added
> > global filtersafety
> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> > 2012-05-17 21:41:28,615 WARN org.apache.hadoop.util.NativeCodeLoader:
> > Unable to load native-hadoop library for your platform... using
> > builtin-java classes where applicable
> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Port
> > returned by webServer.getConnectors()[0].getLocalPort() before open() is
> > -1. Opening the listener on 50030
> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer:
> > listener.getLocalPort() returned 50030
> > webServer.getConnectors()[0].getLocalPort() returned 50030
> > 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Jetty
> > bound to port 50030
> > 2012-05-17 21:41:28,615 INFO org.mortbay.log: jetty-6.1.26
> > 2012-05-17 21:41:28,834 INFO org.mortbay.log: Started
> > SelectChannelConnector@0.0.0.0:50030
> > 2012-05-17 21:41:28,834 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> jvm
> > registered.
> > 2012-05-17 21:41:28,834 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > JobTrackerMetrics registered.
> > 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
> > JobTracker up at: 47111
> > 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
> > JobTracker webserver: 50030
> > 2012-05-17 21:41:29,225 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2012-05-17 21:41:29,772 INFO org.apache.hadoop.mapred.JobHistory:
> Creating
> > DONE folder at file:/C:/cygwin/home/hadoop-1.0.3/logs/history/done
> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker: History
> > server being initialized in embedded mode
> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobHistoryServer:
> > Started job history server at: localhost:50030
> > 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker: Job
> > History Server web address: localhost:50030
> > 2012-05-17 21:41:29,787 INFO
> > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
> > inactive
> > 2012-05-17 21:41:29,990 INFO org.apache.hadoop.mapred.JobTracker:
> > Refreshing hosts information
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
> > Setting the includes file to
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
> > Setting the excludes file to
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
> > Refreshing hosts (include/exclude) list
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker:
> > Decommissioning 0 nodes
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > Responder: starting
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > listener on 47111: starting
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 0 on 47111: starting
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 47111: starting
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 2 on 47111: starting
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 47111: starting
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 4 on 47111: starting
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 5 on 47111: starting
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 6 on 47111: starting
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 7 on 47111: starting
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker:
> Starting
> > RUNNING
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 8 on 47111: starting
> > 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 9 on 47111: starting
> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
> > job_201205172141_0001: nMaps=16 nReduces=1 max=-1
> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker: Job
> > job_201205172141_0001 added successfully for user 'rn13067' to queue
> > 'default'
> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker:
> > Initializing job_201205172141_0001
> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
> > Initializing job_201205172141_0001
> > 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.AuditLogger:
> > USER=rn13067 IP=192.168.0.16 OPERATION=SUBMIT_JOB
> > TARGET=job_201205172141_0001 RESULT=SUCCESS
> > *2012-05-17 21:44:47,084 ERROR org.apache.hadoop.mapred.JobHistory:
> > Failed creating job history log file for job job_201205172141_0001
> > java.io.IOException: Failed to set permissions of path:
> >
> C:\cygwin\home\hadoop-1.0.3\logs\history\job_201205172141_0001_1337305487022_rn13067_grep-search
> > to 0744*
> > at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
> > at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:678)
> > at
> >
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> > at
> >
> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:286)
> > at
> >
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:385)
> > at
> >
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
> > at
> >
> org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:1696)
> > at
> org.apache.hadoop.mapred.JobInProgress$3.run(JobInProgress.java:681)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> > at
> > org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:678)
> > at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207)
> > at
> >
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> >
> > Kindly help me. CYGWIN is in the path. (As some one suggested in some
> > thread).
> >
> >
> >
> >
> > --
> > Warmest Regards,
> >
> > Ravi
> >
>
--
Warmest Regards,
Ravi Shankar
Re: Why this problem is not solved yet ?
Posted by Ravi Prakash <ra...@gmail.com>.
Ravishankar,
If you run $ jps, do you see a TaskTracker process running? Can you please
post the tasktracker logs as well?
On Thu, May 17, 2012 at 8:49 PM, Ravishankar Nair <
ravishankar.nair@gmail.com> wrote:
> Dear experts,
>
> Today is my tenth day working with Hadoop on installing on my windows
> machine. I am trying again and again because , some where someone has
> written that it works on Windows with CYGWIN.(And noone has written that
> Hadoop wont work on Windows). I am attaching my config files.
>
> Kindly help me, if anything can make this work.A feeble and humble request
> to all experts out there.
>
> Here is the error, if you search , you can see thousands have reported
> this and there is no solution I got yet, though I tried all ways possible.
> I am using Windows XP SP3, Hadoop (tried with five versions so far
> including 1.0.3). I am running on a single node.(machine WSUSJXLHRN13067
> IP:- 192.168.0.16)
> When I start Hadoop, no issues in any of the versions
>
> rn13067@WSUSJXLHRN13067 /home/hadoop-1.0.3
> $ bin/start-all.sh
> starting namenode, logging to
> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-namenode-WSUSJXLHRN13067.out
> localhost: starting datanode, logging to
> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-datanode-WSUSJXLHRN13067.out
> localhost: starting secondarynamenode, logging to
> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-secondarynamenode-WSUSJXLHRN13067.out
> starting jobtracker, logging to
> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-jobtracker-WSUSJXLHRN13067.out
> localhost: starting tasktracker, logging to
> /home/hadoop-1.0.3/libexec/../logs/hadoop-SUNDOOP-tasktracker-WSUSJXLHRN13067.out
>
>
>
> When I run the example program, this is what is printed on my console:-
> $ bin/hadoop jar hadoop-examples-1.0.3.jar grep input output 'dfs[a-z.]+'
> 12/05/17 21:44:46 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 12/05/17 21:44:46 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/05/17 21:44:46 INFO mapred.FileInputFormat: Total input paths to
> process : 16
> 12/05/17 21:44:47 INFO mapred.JobClient: Running job: job_201205172141_0001
> 12/05/17 21:44:48 INFO mapred.JobClient: map 0% reduce 0%
>
>
> Now it is HUNG!!. IN most of the versions this is the behaviour.
>
> Here is the log from JOBTRACKER:-
>
>
> 2012-05-17 21:41:28,037 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = WSUSJXLHRN13067/192.168.0.16
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.3
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
> ************************************************************/
> 2012-05-17 21:41:28,147 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2012-05-17 21:41:28,147 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2012-05-17 21:41:28,162 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2012-05-17 21:41:28,162 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics
> system started
> 2012-05-17 21:41:28,209 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> QueueMetrics,q=default registered.
> 2012-05-17 21:41:28,428 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2012-05-17 21:41:28,428 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2012-05-17 21:41:28,428 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Updating the current master key for generating delegation tokens
> 2012-05-17 21:41:28,428 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Starting expired delegation token remover thread,
> tokenRemoverScanInterval=60 min(s)
> 2012-05-17 21:41:28,428 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Updating the current master key for generating delegation tokens
> 2012-05-17 21:41:28,428 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2012-05-17 21:41:28,428 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2012-05-17 21:41:28,444 INFO org.apache.hadoop.mapred.JobTracker: Starting
> jobtracker with owner as rn13067
> 2012-05-17 21:41:28,475 INFO org.apache.hadoop.ipc.Server: Starting
> SocketReader
> 2012-05-17 21:41:28,475 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> RpcDetailedActivityForPort47111 registered.
> 2012-05-17 21:41:28,475 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> RpcActivityForPort47111 registered.
> 2012-05-17 21:41:28,522 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2012-05-17 21:41:28,584 INFO org.apache.hadoop.http.HttpServer: Added
> global filtersafety
> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2012-05-17 21:41:28,615 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50030
> 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50030
> webServer.getConnectors()[0].getLocalPort() returned 50030
> 2012-05-17 21:41:28,615 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50030
> 2012-05-17 21:41:28,615 INFO org.mortbay.log: jetty-6.1.26
> 2012-05-17 21:41:28,834 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50030
> 2012-05-17 21:41:28,834 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
> registered.
> 2012-05-17 21:41:28,834 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> JobTrackerMetrics registered.
> 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker up at: 47111
> 2012-05-17 21:41:28,850 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker webserver: 50030
> 2012-05-17 21:41:29,225 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> up the system directory
> 2012-05-17 21:41:29,772 INFO org.apache.hadoop.mapred.JobHistory: Creating
> DONE folder at file:/C:/cygwin/home/hadoop-1.0.3/logs/history/done
> 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker: History
> server being initialized in embedded mode
> 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobHistoryServer:
> Started job history server at: localhost:50030
> 2012-05-17 21:41:29,787 INFO org.apache.hadoop.mapred.JobTracker: Job
> History Server web address: localhost:50030
> 2012-05-17 21:41:29,787 INFO
> org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
> inactive
> 2012-05-17 21:41:29,990 INFO org.apache.hadoop.mapred.JobTracker:
> Refreshing hosts information
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
> Setting the includes file to
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
> Setting the excludes file to
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker:
> Decommissioning 0 nodes
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 47111: starting
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 47111: starting
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 47111: starting
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 47111: starting
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 47111: starting
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 4 on 47111: starting
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 47111: starting
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 6 on 47111: starting
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 7 on 47111: starting
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.mapred.JobTracker: Starting
> RUNNING
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 8 on 47111: starting
> 2012-05-17 21:41:30,006 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 9 on 47111: starting
> 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
> job_201205172141_0001: nMaps=16 nReduces=1 max=-1
> 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker: Job
> job_201205172141_0001 added successfully for user 'rn13067' to queue
> 'default'
> 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobTracker:
> Initializing job_201205172141_0001
> 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.JobInProgress:
> Initializing job_201205172141_0001
> 2012-05-17 21:44:47,069 INFO org.apache.hadoop.mapred.AuditLogger:
> USER=rn13067 IP=192.168.0.16 OPERATION=SUBMIT_JOB
> TARGET=job_201205172141_0001 RESULT=SUCCESS
> *2012-05-17 21:44:47,084 ERROR org.apache.hadoop.mapred.JobHistory:
> Failed creating job history log file for job job_201205172141_0001
> java.io.IOException: Failed to set permissions of path:
> C:\cygwin\home\hadoop-1.0.3\logs\history\job_201205172141_0001_1337305487022_rn13067_grep-search
> to 0744*
> at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:678)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at
> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:286)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:385)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
> at
> org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:1696)
> at org.apache.hadoop.mapred.JobInProgress$3.run(JobInProgress.java:681)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at
> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:678)
> at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207)
> at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Kindly help me. CYGWIN is in the path. (As some one suggested in some
> thread).
>
>
>
>
> --
> Warmest Regards,
>
> Ravi
>