You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by npriya <n....@gmail.com> on 2010/08/25 12:06:48 UTC
Error regarding the submission of jobs in cluster.
I formed a hadoop cluster with 7 m/c .I am trying to do a small change in
the fairscheduler that allow large jobs to finish of first before small
one,and also trying to backfill small jobs in between the delay of two large
jobs.But after submission such an error is coming. What is the solution??
root@cluster3 hadoop-0.20.1]# bin/hadoop jar hadoop-0.20.1-examples.jar
sleep -m 3 -r 3 -mt 1200 -rt 1300
10/08/19 15:24:12 INFO mapred.JobClient: Running job: job_201008191510_0004
10/08/19 15:24:13 INFO mapred.JobClient: map 0% reduce 0%
10/08/19 15:24:18 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_m_000004_0, Status : FAILED
Error initializing attempt_201008191510_0004_m_000004_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:18 WARN mapred.JobClient: Error reading task
outputhttp://cluster7:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000004_0&filter=stdout
10/08/19 15:24:18 WARN mapred.JobClient: Error reading task
outputhttp://cluster7:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000004_0&filter=stderr
10/08/19 15:24:21 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_r_000004_0, Status : FAILED
Error initializing attempt_201008191510_0004_r_000004_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:21 WARN mapred.JobClient: Error reading task
outputhttp://cluster7:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_r_000004_0&filter=stdout
10/08/19 15:24:21 WARN mapred.JobClient: Error reading task
outputhttp://cluster7:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_r_000004_0&filter=stderr
10/08/19 15:24:27 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_m_000004_1, Status : FAILED
Error initializing attempt_201008191510_0004_m_000004_1:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:27 WARN mapred.JobClient: Error reading task
outputhttp://cluster6:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000004_1&filter=stdout
10/08/19 15:24:27 WARN mapred.JobClient: Error reading task
outputhttp://cluster6:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000004_1&filter=stderr
10/08/19 15:24:30 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_r_000004_1, Status : FAILED
Error initializing attempt_201008191510_0004_r_000004_1:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:30 WARN mapred.JobClient: Error reading task
outputhttp://cluster6:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_r_000004_1&filter=stdout
10/08/19 15:24:30 WARN mapred.JobClient: Error reading task
outputhttp://cluster6:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_r_000004_1&filter=stderr
10/08/19 15:24:45 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_m_000000_0, Status : FAILED
Error initializing attempt_201008191510_0004_m_000000_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:45 WARN mapred.JobClient: Error reading task
outputhttp://cluster2:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000000_0&filter=stdout
10/08/19 15:24:45 WARN mapred.JobClient: Error reading task
outputhttp://cluster2:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000000_0&filter=stderr
10/08/19 15:24:45 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_m_000001_0, Status : FAILED
Error initializing attempt_201008191510_0004_m_000001_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:45 WARN mapred.JobClient: Error reading task
outputhttp://cluster7:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000001_0&filter=stdout
10/08/19 15:24:45 WARN mapred.JobClient: Error reading task
outputhttp://cluster7:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000001_0&filter=stderr
10/08/19 15:24:45 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_m_000002_0, Status : FAILED
Error initializing attempt_201008191510_0004_m_000002_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:45 WARN mapred.JobClient: Error reading task
outputhttp://cluster6:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000002_0&filter=stdout
10/08/19 15:24:45 WARN mapred.JobClient: Error reading task
outputhttp://cluster6:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000002_0&filter=stderr
10/08/19 15:24:51 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_m_000002_1, Status : FAILED
Error initializing attempt_201008191510_0004_m_000002_1:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:51 WARN mapred.JobClient: Error reading task
outputhttp://cluster2:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000002_1&filter=stdout
10/08/19 15:24:51 WARN mapred.JobClient: Error reading task
outputhttp://cluster2:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000002_1&filter=stderr
10/08/19 15:24:51 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_m_000000_1, Status : FAILED
Error initializing attempt_201008191510_0004_m_000000_1:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:51 WARN mapred.JobClient: Error reading task
outputhttp://cluster7:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000000_1&filter=stdout
10/08/19 15:24:51 WARN mapred.JobClient: Error reading task
outputhttp://cluster7:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000000_1&filter=stderr
10/08/19 15:24:51 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_m_000001_1, Status : FAILED
Error initializing attempt_201008191510_0004_m_000001_1:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:51 WARN mapred.JobClient: Error reading task
outputhttp://cluster4:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000001_1&filter=stdout
10/08/19 15:24:51 WARN mapred.JobClient: Error reading task
outputhttp://cluster4:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000001_1&filter=stderr
10/08/19 15:24:57 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_m_000001_2, Status : FAILED
Error initializing attempt_201008191510_0004_m_000001_2:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:57 WARN mapred.JobClient: Error reading task
outputhttp://cluster2:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000001_2&filter=stdout
10/08/19 15:24:57 WARN mapred.JobClient: Error reading task
outputhttp://cluster2:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000001_2&filter=stderr
10/08/19 15:24:57 INFO mapred.JobClient: Task Id :
attempt_201008191510_0004_m_000000_2, Status : FAILED
Error initializing attempt_201008191510_0004_m_000000_2:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for taskTracker/jobcache/job_201008191510_0004/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
10/08/19 15:24:57 WARN mapred.JobClient: Error reading task
outputhttp://cluster6:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000000_2&filter=stdout
10/08/19 15:24:57 WARN mapred.JobClient: Error reading task
outputhttp://cluster6:50060/tasklog?plaintext=true&taskid=attempt_201008191510_0004_m_000000_2&filter=stderr
10/08/19 15:24:58 INFO mapred.JobClient: map 33% reduce 0%
10/08/19 15:25:04 INFO mapred.JobClient: map 100% reduce 0%
10/08/19 15:25:13 INFO mapred.JobClient: map 100% reduce 66%
10/08/19 15:25:22 INFO mapred.JobClient: map 100% reduce 100%
10/08/19 15:25:24 INFO mapred.JobClient: Job complete: job_201008191510_0004
10/08/19 15:25:24 INFO mapred.JobClient: Counters: 15
10/08/19 15:25:24 INFO mapred.JobClient: Job Counters
10/08/19 15:25:24 INFO mapred.JobClient: Launched reduce tasks=3
10/08/19 15:25:24 INFO mapred.JobClient: Launched map tasks=11
10/08/19 15:25:24 INFO mapred.JobClient: FileSystemCounters
10/08/19 15:25:24 INFO mapred.JobClient: FILE_BYTES_READ=720
10/08/19 15:25:24 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1716
10/08/19 15:25:24 INFO mapred.JobClient: Map-Reduce Framework
10/08/19 15:25:24 INFO mapred.JobClient: Reduce input groups=39
10/08/19 15:25:24 INFO mapred.JobClient: Combine output records=0
10/08/19 15:25:24 INFO mapred.JobClient: Map input records=36
10/08/19 15:25:24 INFO mapred.JobClient: Reduce shuffle bytes=588
10/08/19 15:25:24 INFO mapred.JobClient: Reduce output records=0
10/08/19 15:25:24 INFO mapred.JobClient: Spilled Records=234
10/08/19 15:25:24 INFO mapred.JobClient: Map output bytes=468
10/08/19 15:25:24 INFO mapred.JobClient: Map input bytes=36
10/08/19 15:25:24 INFO mapred.JobClient: Combine input records=0
10/08/19 15:25:24 INFO mapred.JobClient: Map output records=117
10/08/19 15:25:24 INFO mapred.JobClient: Reduce input records=0
--
View this message in context: http://old.nabble.com/Error-regarding-the-submission-of-jobs-in-cluster.-tp29530606p29530606.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.