You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Corbett Martin <co...@nhin.com> on 2013/01/23 18:28:51 UTC
Trouble starting up Task Tracker
Question
We're trying out Cloudera Manager and CDH4 in a Clustered deployment and having trouble getting the Task Trackers to start up.
The error says (full stacktrace below)
2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local directories are writable
at org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
What directories is it referring to? We have the following specified on the local file system.
ls -al /data/1/mapred/
drwxr-xr-x 4 mapred hadoop 4096 Jan 15 16:37 .
drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
ls -al /data/2/mapred/
drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
ls -al /data/3/mapred/
drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
When we try to start the Map Reduce service from Cloudera Manager the Task Trackers fail to start. The Job Tracker starts fine. Here's the error from the log file.
************************************************************/
2013-01-23 10:48:35,951 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = foo-dev-arch-02/10.204.12.63
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.0.0-mr1-cdh4.1.1
STARTUP_MSG: classpath = /var/run/cloudera-scm-agent/process/499-mapreduce-TASKTRACKER:/usr/java/jdk1.6.0_30/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-4.1.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/hue-plugins-2.1.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-auth.jar::
STARTUP_MSG: build = file:///data/1/jenkins/workspace/generic-package-sles64-11/topdir/BUILD/hadoop-2.0.0-mr1-cdh4.1.1 -r Unknown; compiled by 'jenkins' on Tue Oct 16 11:50:57 PDT 2012
************************************************************/
2013-01-23 10:48:37,095 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2013-01-23 10:48:37,189 INFO org.apache.hadoop.http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context task
2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2013-01-23 10:48:37,441 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker local dir /data/1/mapred/local error File /data/1/mapred/local does not exist, removing from local dirs
2013-01-23 10:48:37,442 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker local dir /data/2/mapred/local error File /data/2/mapred/local does not exist, removing from local dirs
2013-01-23 10:48:37,443 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker local dir /data/3/mapred/local error File /data/3/mapred/local does not exist, removing from local dirs
2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local directories are writable
at org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
2013-01-23 10:48:37,446 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at foo-dev-arch-02/10.204.12.63
************************************************************/
Thanks in advance
~Corbett Martin
Software Architect
AbsoluteAR Accounts Receivable Services - An NHIN Solution
________________________________
This message and its contents (to include attachments) are the property of National Health Systems, Inc. and may contain confidential and proprietary information. This email and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed. You are hereby notified that any unauthorized disclosure, copying, or distribution of this message, or the taking of any unauthorized action based on information contained herein is strictly prohibited. Unauthorized use of information contained herein may subject you to civil and criminal prosecution and penalties. If you are not the intended recipient, you should delete this message immediately and notify the sender immediately by telephone or by replying to this transmission.
Re: Trouble starting up Task Tracker
Posted by Harsh J <ha...@cloudera.com>.
This is the problem:
drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
Your /data/1 directory seems to be owned by hdfs and restricted only
to it (700). I'm not sure this is necessary and you can perhaps make
it 755 at least.
Or perhaps what you may have is a misconfig wherein you've set your DN
config of dfs.datanode.data.dir to a set of /data/{1..n} instead of
the regular /data/{1..n}/dn or so. The DN has a feature where it
applies dfs.datanode.data.dir.perm (700) to its configured disks for
security - thereby marking all of your mount entry points as 700 and
causing trouble with the mapred.local.dir config of
/data/{1..n}/mapred which resides under it.
On Wed, Jan 23, 2013 at 10:58 PM, Corbett Martin <co...@nhin.com> wrote:
> Question
>
>
>
> We’re trying out Cloudera Manager and CDH4 in a Clustered deployment and
> having trouble getting the Task Trackers to start up.
>
>
>
> The error says (full stacktrace below)
>
> 2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because
> org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local
> directories are writable
>
> at
> org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
>
> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
>
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
>
>
>
> What directories is it referring to? We have the following specified on the
> local file system.
>
>
>
> ls -al /data/1/mapred/
>
> drwxr-xr-x 4 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/2/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/3/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> When we try to start the Map Reduce service from Cloudera Manager the Task
> Trackers fail to start. The Job Tracker starts fine. Here’s the error from
> the log file.
>
>
>
> ************************************************************/
>
> 2013-01-23 10:48:35,951 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
>
> /************************************************************
>
> STARTUP_MSG: Starting TaskTracker
>
> STARTUP_MSG: host = foo-dev-arch-02/10.204.12.63
>
> STARTUP_MSG: args = []
>
> STARTUP_MSG: version = 2.0.0-mr1-cdh4.1.1
>
> STARTUP_MSG: classpath =
> /var/run/cloudera-scm-agent/process/499-mapreduce-TASKTRACKER:/usr/java/jdk1.6.0_30/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-4.1.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/hue-plugins-2.1.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-auth.jar::
>
> STARTUP_MSG: build =
> file:///data/1/jenkins/workspace/generic-package-sles64-11/topdir/BUILD/hadoop-2.0.0-mr1-cdh4.1.1
> -r Unknown; compiled by 'jenkins' on Tue Oct 16 11:50:57 PDT 2012
>
> ************************************************************/
>
> 2013-01-23 10:48:37,095 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
>
> 2013-01-23 10:48:37,189 INFO org.apache.hadoop.http.HttpServer: Added global
> filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context task
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
>
> 2013-01-23 10:48:37,441 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/1/mapred/local error File /data/1/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,442 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/2/mapred/local error File /data/2/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,443 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/3/mapred/local error File /data/3/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because
> org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local
> directories are writable
>
> at
> org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
>
> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
>
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
>
>
>
> 2013-01-23 10:48:37,446 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down TaskTracker at foo-dev-arch-02/10.204.12.63
>
> ************************************************************/
>
>
>
> Thanks in advance
>
>
>
> ~Corbett Martin
>
> Software Architect
>
> AbsoluteAR Accounts Receivable Services - An NHIN Solution
>
>
>
>
> ________________________________
> This message and its contents (to include attachments) are the property of
> National Health Systems, Inc. and may contain confidential and proprietary
> information. This email and any files transmitted with it are intended
> solely for the use of the individual or entity to whom they are addressed.
> You are hereby notified that any unauthorized disclosure, copying, or
> distribution of this message, or the taking of any unauthorized action based
> on information contained herein is strictly prohibited. Unauthorized use of
> information contained herein may subject you to civil and criminal
> prosecution and penalties. If you are not the intended recipient, you should
> delete this message immediately and notify the sender immediately by
> telephone or by replying to this transmission.
--
Harsh J
Re: Trouble starting up Task Tracker
Posted by Harsh J <ha...@cloudera.com>.
This is the problem:
drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
Your /data/1 directory seems to be owned by hdfs and restricted only
to it (700). I'm not sure this is necessary and you can perhaps make
it 755 at least.
Or perhaps what you may have is a misconfig wherein you've set your DN
config of dfs.datanode.data.dir to a set of /data/{1..n} instead of
the regular /data/{1..n}/dn or so. The DN has a feature where it
applies dfs.datanode.data.dir.perm (700) to its configured disks for
security - thereby marking all of your mount entry points as 700 and
causing trouble with the mapred.local.dir config of
/data/{1..n}/mapred which resides under it.
On Wed, Jan 23, 2013 at 10:58 PM, Corbett Martin <co...@nhin.com> wrote:
> Question
>
>
>
> We’re trying out Cloudera Manager and CDH4 in a Clustered deployment and
> having trouble getting the Task Trackers to start up.
>
>
>
> The error says (full stacktrace below)
>
> 2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because
> org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local
> directories are writable
>
> at
> org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
>
> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
>
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
>
>
>
> What directories is it referring to? We have the following specified on the
> local file system.
>
>
>
> ls -al /data/1/mapred/
>
> drwxr-xr-x 4 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/2/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/3/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> When we try to start the Map Reduce service from Cloudera Manager the Task
> Trackers fail to start. The Job Tracker starts fine. Here’s the error from
> the log file.
>
>
>
> ************************************************************/
>
> 2013-01-23 10:48:35,951 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
>
> /************************************************************
>
> STARTUP_MSG: Starting TaskTracker
>
> STARTUP_MSG: host = foo-dev-arch-02/10.204.12.63
>
> STARTUP_MSG: args = []
>
> STARTUP_MSG: version = 2.0.0-mr1-cdh4.1.1
>
> STARTUP_MSG: classpath =
> /var/run/cloudera-scm-agent/process/499-mapreduce-TASKTRACKER:/usr/java/jdk1.6.0_30/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-4.1.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/hue-plugins-2.1.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-auth.jar::
>
> STARTUP_MSG: build =
> file:///data/1/jenkins/workspace/generic-package-sles64-11/topdir/BUILD/hadoop-2.0.0-mr1-cdh4.1.1
> -r Unknown; compiled by 'jenkins' on Tue Oct 16 11:50:57 PDT 2012
>
> ************************************************************/
>
> 2013-01-23 10:48:37,095 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
>
> 2013-01-23 10:48:37,189 INFO org.apache.hadoop.http.HttpServer: Added global
> filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context task
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
>
> 2013-01-23 10:48:37,441 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/1/mapred/local error File /data/1/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,442 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/2/mapred/local error File /data/2/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,443 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/3/mapred/local error File /data/3/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because
> org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local
> directories are writable
>
> at
> org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
>
> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
>
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
>
>
>
> 2013-01-23 10:48:37,446 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down TaskTracker at foo-dev-arch-02/10.204.12.63
>
> ************************************************************/
>
>
>
> Thanks in advance
>
>
>
> ~Corbett Martin
>
> Software Architect
>
> AbsoluteAR Accounts Receivable Services - An NHIN Solution
>
>
>
>
> ________________________________
> This message and its contents (to include attachments) are the property of
> National Health Systems, Inc. and may contain confidential and proprietary
> information. This email and any files transmitted with it are intended
> solely for the use of the individual or entity to whom they are addressed.
> You are hereby notified that any unauthorized disclosure, copying, or
> distribution of this message, or the taking of any unauthorized action based
> on information contained herein is strictly prohibited. Unauthorized use of
> information contained herein may subject you to civil and criminal
> prosecution and penalties. If you are not the intended recipient, you should
> delete this message immediately and notify the sender immediately by
> telephone or by replying to this transmission.
--
Harsh J
Re: Trouble starting up Task Tracker
Posted by Harsh J <ha...@cloudera.com>.
This is the problem:
drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
Your /data/1 directory seems to be owned by hdfs and restricted only
to it (700). I'm not sure this is necessary and you can perhaps make
it 755 at least.
Or perhaps what you may have is a misconfig wherein you've set your DN
config of dfs.datanode.data.dir to a set of /data/{1..n} instead of
the regular /data/{1..n}/dn or so. The DN has a feature where it
applies dfs.datanode.data.dir.perm (700) to its configured disks for
security - thereby marking all of your mount entry points as 700 and
causing trouble with the mapred.local.dir config of
/data/{1..n}/mapred which resides under it.
On Wed, Jan 23, 2013 at 10:58 PM, Corbett Martin <co...@nhin.com> wrote:
> Question
>
>
>
> We’re trying out Cloudera Manager and CDH4 in a Clustered deployment and
> having trouble getting the Task Trackers to start up.
>
>
>
> The error says (full stacktrace below)
>
> 2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because
> org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local
> directories are writable
>
> at
> org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
>
> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
>
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
>
>
>
> What directories is it referring to? We have the following specified on the
> local file system.
>
>
>
> ls -al /data/1/mapred/
>
> drwxr-xr-x 4 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/2/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/3/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> When we try to start the Map Reduce service from Cloudera Manager the Task
> Trackers fail to start. The Job Tracker starts fine. Here’s the error from
> the log file.
>
>
>
> ************************************************************/
>
> 2013-01-23 10:48:35,951 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
>
> /************************************************************
>
> STARTUP_MSG: Starting TaskTracker
>
> STARTUP_MSG: host = foo-dev-arch-02/10.204.12.63
>
> STARTUP_MSG: args = []
>
> STARTUP_MSG: version = 2.0.0-mr1-cdh4.1.1
>
> STARTUP_MSG: classpath =
> /var/run/cloudera-scm-agent/process/499-mapreduce-TASKTRACKER:/usr/java/jdk1.6.0_30/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-4.1.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/hue-plugins-2.1.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-auth.jar::
>
> STARTUP_MSG: build =
> file:///data/1/jenkins/workspace/generic-package-sles64-11/topdir/BUILD/hadoop-2.0.0-mr1-cdh4.1.1
> -r Unknown; compiled by 'jenkins' on Tue Oct 16 11:50:57 PDT 2012
>
> ************************************************************/
>
> 2013-01-23 10:48:37,095 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
>
> 2013-01-23 10:48:37,189 INFO org.apache.hadoop.http.HttpServer: Added global
> filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context task
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
>
> 2013-01-23 10:48:37,441 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/1/mapred/local error File /data/1/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,442 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/2/mapred/local error File /data/2/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,443 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/3/mapred/local error File /data/3/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because
> org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local
> directories are writable
>
> at
> org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
>
> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
>
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
>
>
>
> 2013-01-23 10:48:37,446 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down TaskTracker at foo-dev-arch-02/10.204.12.63
>
> ************************************************************/
>
>
>
> Thanks in advance
>
>
>
> ~Corbett Martin
>
> Software Architect
>
> AbsoluteAR Accounts Receivable Services - An NHIN Solution
>
>
>
>
> ________________________________
> This message and its contents (to include attachments) are the property of
> National Health Systems, Inc. and may contain confidential and proprietary
> information. This email and any files transmitted with it are intended
> solely for the use of the individual or entity to whom they are addressed.
> You are hereby notified that any unauthorized disclosure, copying, or
> distribution of this message, or the taking of any unauthorized action based
> on information contained herein is strictly prohibited. Unauthorized use of
> information contained herein may subject you to civil and criminal
> prosecution and penalties. If you are not the intended recipient, you should
> delete this message immediately and notify the sender immediately by
> telephone or by replying to this transmission.
--
Harsh J
Re: Trouble starting up Task Tracker
Posted by Harsh J <ha...@cloudera.com>.
This is the problem:
drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
Your /data/1 directory seems to be owned by hdfs and restricted only
to it (700). I'm not sure this is necessary and you can perhaps make
it 755 at least.
Or perhaps what you may have is a misconfig wherein you've set your DN
config of dfs.datanode.data.dir to a set of /data/{1..n} instead of
the regular /data/{1..n}/dn or so. The DN has a feature where it
applies dfs.datanode.data.dir.perm (700) to its configured disks for
security - thereby marking all of your mount entry points as 700 and
causing trouble with the mapred.local.dir config of
/data/{1..n}/mapred which resides under it.
On Wed, Jan 23, 2013 at 10:58 PM, Corbett Martin <co...@nhin.com> wrote:
> Question
>
>
>
> We’re trying out Cloudera Manager and CDH4 in a Clustered deployment and
> having trouble getting the Task Trackers to start up.
>
>
>
> The error says (full stacktrace below)
>
> 2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because
> org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local
> directories are writable
>
> at
> org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
>
> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
>
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
>
>
>
> What directories is it referring to? We have the following specified on the
> local file system.
>
>
>
> ls -al /data/1/mapred/
>
> drwxr-xr-x 4 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/2/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/3/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs hdfs 4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> When we try to start the Map Reduce service from Cloudera Manager the Task
> Trackers fail to start. The Job Tracker starts fine. Here’s the error from
> the log file.
>
>
>
> ************************************************************/
>
> 2013-01-23 10:48:35,951 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
>
> /************************************************************
>
> STARTUP_MSG: Starting TaskTracker
>
> STARTUP_MSG: host = foo-dev-arch-02/10.204.12.63
>
> STARTUP_MSG: args = []
>
> STARTUP_MSG: version = 2.0.0-mr1-cdh4.1.1
>
> STARTUP_MSG: classpath =
> /var/run/cloudera-scm-agent/process/499-mapreduce-TASKTRACKER:/usr/java/jdk1.6.0_30/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-4.1.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/hue-plugins-2.1.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-auth.jar::
>
> STARTUP_MSG: build =
> file:///data/1/jenkins/workspace/generic-package-sles64-11/topdir/BUILD/hadoop-2.0.0-mr1-cdh4.1.1
> -r Unknown; compiled by 'jenkins' on Tue Oct 16 11:50:57 PDT 2012
>
> ************************************************************/
>
> 2013-01-23 10:48:37,095 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
>
> 2013-01-23 10:48:37,189 INFO org.apache.hadoop.http.HttpServer: Added global
> filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context task
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
>
> 2013-01-23 10:48:37,441 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/1/mapred/local error File /data/1/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,442 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/2/mapred/local error File /data/2/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,443 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/3/mapred/local error File /data/3/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because
> org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local
> directories are writable
>
> at
> org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
>
> at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
>
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
>
>
>
> 2013-01-23 10:48:37,446 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down TaskTracker at foo-dev-arch-02/10.204.12.63
>
> ************************************************************/
>
>
>
> Thanks in advance
>
>
>
> ~Corbett Martin
>
> Software Architect
>
> AbsoluteAR Accounts Receivable Services - An NHIN Solution
>
>
>
>
> ________________________________
> This message and its contents (to include attachments) are the property of
> National Health Systems, Inc. and may contain confidential and proprietary
> information. This email and any files transmitted with it are intended
> solely for the use of the individual or entity to whom they are addressed.
> You are hereby notified that any unauthorized disclosure, copying, or
> distribution of this message, or the taking of any unauthorized action based
> on information contained herein is strictly prohibited. Unauthorized use of
> information contained herein may subject you to civil and criminal
> prosecution and penalties. If you are not the intended recipient, you should
> delete this message immediately and notify the sender immediately by
> telephone or by replying to this transmission.
--
Harsh J