TaskTracker's tasksMaxMem is not set. TaskMemoryManager is disabled.

[Ilana Heintz <he...@osu.edu>]

Hello,

I see in the archives that last month somebody had a question along these 
lines, but there didn't seem to be a real resolution.  I am unable to run 
any processes due to some issue with the task tracker that I don't 
understand.

Command line interactions:

$ bin/hadoop dfs -mkdir input
$ bin/hadoop dfs -copyFromLocal ~/inputfiles.* input
$ bin/hadoop jar hadoop-0.19.0-examples.jar wordcount input output
09/10/01 13:02:56 INFO mapred.FileInputFormat: Total input paths to 
process : 10
09/10/01 13:02:57 INFO mapred.JobClient: Running job: job_200910011254_0003
09/10/01 13:02:58 INFO mapred.JobClient:  map 0% reduce 0%
09/10/01 13:03:07 INFO mapred.JobClient: Task Id : 
attempt_200910011254_0003_m_000045_0, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:424)

09/10/01 13:03:07 WARN mapred.JobClient: Error reading task 
outputhttp://tile.cse.ohio-state.edu:50060/tasklog?plaintext=true&taskid=attempt_200910011254_0003_m_000045_0&filter=stdout
09/10/01 13:03:07 WARN mapred.JobClient: Error reading task 
outputhttp://tile.cse.ohio-state.edu:50060/tasklog?plaintext=true&taskid=attempt_200910011254_0003_m_000045_0&filter=stderr
09/10/01 13:03:10 INFO mapred.JobClient: Task Id :

With subsequent attempts that give the same error.

>From the log file, 
hadoop-bromberg-tasktracker-roof.cse.ohio-state.edu.log:

2009-10-01 12:37:48,242 INFO org.apache.hadoop.mapred.TaskTracker: 
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG:   host = roof.cse.ohio-state.edu/164.107.116.105
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.19.1-dev
STARTUP_MSG:   build =  -r ; compiled by 'cbrew' on Tue Jan 20 15:22:44 EST 2009
************************************************************/
2009-10-01 12:37:49,236 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
2009-10-01 12:37:49,261 INFO org.mortbay.util.Credential: Checking 
Resource aliases
2009-10-01 12:37:50,437 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebAppli
cationHandler@149d886
2009-10-01 12:37:50,567 INFO org.mortbay.util.Container: Started WebApplicationContext[/static,/sta
tic]
2009-10-01 12:37:50,858 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebAppli
cationHandler@19f3736
2009-10-01 12:37:50,859 INFO org.mortbay.util.Container: Started WebApplicationContext[/logs,/logs]
2009-10-01 12:37:51,127 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebAppli
cationHandler@c8376b
2009-10-01 12:37:51,138 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/]
2009-10-01 12:37:51,148 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:500
60
2009-10-01 12:37:51,148 INFO org.mortbay.util.Container: Started org.mortbay.jetty.Server@dc6a77
2009-10-01 12:37:51,163 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics wit
h processName=TaskTracker, sessionId=2009-10-01 12:37:51,215 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: 
Initializing RPC Metrics with hostName=TaskTracker, port=43163
2009-10-01 12:37:51,347 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2009-10-01 12:37:51,350 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 43163: starting
2009-10-01 12:37:51,352 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 43163: starting
2009-10-01 12:37:51,354 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 43163: starting
2009-10-01 12:37:51,354 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 43163: starting
2009-10-01 12:37:51,362 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost.loc
aldomain/127.0.0.1:43163
2009-10-01 12:37:51,362 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_roof.cs
e.ohio-state.edu:localhost.localdomain/127.0.0.1:43163
2009-10-01 12:37:51,363 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 43163: starting
2009-10-01 12:37:51,473 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetc
her for all reduce tasks on tracker_roof.cse.ohio-state.edu:localhost.localdomain/127.0.0.1:43163
2009-10-01 12:37:51,476 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memor
y = 10485760
2009-10-01 12:37:51,483 INFO org.apache.hadoop.mapred.TaskTracker: 
TaskTracker's tasksMaxMem is not set. TaskMemoryManager is disabled.
2009-10-01 12:39:31,965 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
  attempt_200910011237_0001_m_000045_0
2009-10-01 12:39:31,974 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_20091
0011237_0001_m_000045_0
2009-10-01 12:39:31,974 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free sl
ots : 2 and trying to launch attempt_200910011237_0001_m_000045_0
2009-10-01 12:39:32,750 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
jvm_200910011237_0001_m_-834551651
2009-10-01 12:39:32,751 INFO org.apache.hadoop.mapred.JvmManager: JVM 
Runner jvm_200910011237_0001_m_-834551651 spawned.
2009-10-01 12:39:32,773 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_200910011237_0001_m_-83
4551651 exited. Number of tasks it ran: 0
2009-10-01 12:39:32,776 WARN org.apache.hadoop.mapred.TaskRunner: attempt_200910011237_0001_m_00004
5_0 Child Error
java.io.IOException: Task process exit with nonzero status of 1.
         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:424)
2009-10-01 12:39:35,783 INFO org.apache.hadoop.mapred.TaskRunner: 
attempt_200910011237_0001_m_000045_0 done; removing files.

I believe that's where it starts to repeat.  Similar log files for the 
other nodes in the cloud.

I have looked in the conf/ files for a place to set the tasksMaxMem but 
can only find in hadoop-default.xml:
<property>
   <name>mapred.tasktracker.taskmemorymanager.monitoring-interval</name>
   <value>5000</value>
</property>

I am using hadoop-0.19.0.  I had a working system a month ago, but after a 
complete system reboot, I've had some issues with hadoop.  Everything 
seems to go okay now with copying data in/out of the hdfs, but I can't run 
any processes.  When I do jps on the namenode (roof), I get:
5417 DataNode
5287 NameNode
5779 TaskTracker
5653 JobTracker
5578 SecondaryNameNode
6566 Jps

So it seems like everything loads up fine.
Any pointers on how to fix the tasktracker problem? Or is there a hidden 
problem here?

Thanks,
Ilana

Ilana Heintz
Department of Linguistics
Ohio State University
http://www.ling.ohio-state.edu/~bromberg