You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vinod Gupta Tankala <tv...@readypulse.com> on 2011/09/29 00:21:56 UTC
how to set mapred.system.dir?
Hi,
I am trying to setup a test system to host a distributed hbase installation.
No matter what I do, I get the below errors.
2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery fo
r block null bad datanode[0] nodes == null
2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Could not get
blo
ck locations. Source file "/tmp/mapred/system/jobtracker.info" - Aborting...
2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker: Writing to
fil
e hdfs://
ec2-184-73-22-146.compute-1.amazonaws.com/tmp/mapred/system/jobtracker.
info failed!
2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker: FileSystem
is
not ready yet!
2011-09-28 22:17:26,292 WARN org.apache.hadoop.mapred.JobTracker: Failed to
init
ialize recovery manager.
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/tmp/mapred/sys
tem/jobtracker.info could only be replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc
k(FSNamesystem.java:1417)
....
this is how i setup my config -
core-site.xml -
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
</property>
</configuration>
hdfs-site.xml -
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/tmp/hbase</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/tmp/hbase</value>
</property>
</configuration>
mapred-site.xml -
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>ec2-184-73-22-146.compute-1.amazonaws.com:9001</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/tmp/mapred_tmp</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>10</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>2</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/tmp/mapred/system/</value>
</property>
</configuration>
i know that i am missing something really basic but not sure what it is. the
documentation says mapred.system.dir should be globally accessible. how do i
achieve that?
thanks
vinod
Re: how to set mapred.system.dir?
Posted by Vinod Gupta Tankala <tv...@readypulse.com>.
Thanks Dejo.
I suspected that running distributed mode with single host is going to be
problematic. so i reverted the settings to pseudo-distributed and deleted
everything and did a reformat of the namenode. now hdfs starts fine and i
can see a live node in web console on port 50030.
but when i start a mapred example task (grep-search one) after creating
input dir in dfs, it fails with these kind of errors -
2011-09-29 22:12:12,477 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner c
onstructed JVM ID: jvm_201109292200_0004_m_-1121734218
2011-09-29 22:12:12,477 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm
_201109292200_0004_m_-1121734218 spawned.
2011-09-29 22:12:12,480 INFO org.apache.hadoop.mapred.TaskController:
Writing co
mmands to /media/ephemeral0/hadoop/mapred/local
/ttprivate/taskTracker/ec2-user/jobcache/job_201109292200_0004/attempt_20110
9292200_0004_m_000015_2/taskjvm.sh
2011-09-29 22:12:12,528 WARN org.apache.hadoop.mapred.DefaultTaskController:
Exi
t code from task is : 127
2011-09-29 22:12:12,529 INFO org.apache.hadoop.mapred.DefaultTaskController:
Out
put from DefaultTaskController's launchTask follows:
2011-09-29 22:12:12,529 INFO org.apache.hadoop.mapred.TaskController:
2011-09-29 22:12:12,529 INFO org.apache.hadoop.mapred.JvmManager: JVM Not
killed
jvm_201109292200_0004_m_-1121734218 but just removed
2011-09-29 22:12:12,530 WARN org.apache.hadoop.mapred.TaskRunner:
attempt_201109
292200_0004_m_000015_2 : Child Error
java.io.IOException: Task process exit with nonzero status of 127.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
now i wonder what could be the reason for this.
thanks
vinod
On Thu, Sep 29, 2011 at 2:55 PM, Dejan Menges <de...@gmail.com>wrote:
> Hi Vinod,
>
> This sounds to me like a DNS issue. I have the same thing on development
> environments, and couple of things that made me big headache were:
>
> - Inconsistency of DNS and machine hostname, if not used localhost
> - On some systems in /etc/hosts you have localhost defined for both IPv4
> and
> IPv6 (MacOS is one example) where you need to comment IPv6.
> - When I don't want to use localhost at all, I comment it out totally in
> /etc/hosts
>
> So, the approach I have in this case is to, prior starting doing anything
> with HBase, stop everything, clean the logs, format namenode, and start
> Hadoop only. Monitor logs, what won't be problem in this case as that's
> stand alone installation, to see if HDFS get up OK. You can also try coping
> file from and to your HDFS to be sure HDFS is working fine. Then you can
> start MapReduce (JobTracker and TaskTrackers) and later on HBase.
>
> I hope this will help :)
>
> Regards,
> Dejo
>
> On Thu, Sep 29, 2011 at 8:06 PM, Vinod Gupta Tankala
> <tv...@readypulse.com>wrote:
>
> > Thanks Dejo for pointing that out. I realized that earlier and fixed it.
> > But
> > I still hit the same problem.
> >
> > In my case, I only have a single host for now. But I am still trying to
> do
> > a
> > distributed setup by listing the machine itself as a slave in config and
> > not
> > using localhost anywhere. Does this even work? if not, I can try spending
> > more time on pseudo-distributed setup for now.
> >
> > thanks
> >
> >
> > On Thu, Sep 29, 2011 at 4:48 AM, Dejan Menges <dejan.menges@gmail.com
> > >wrote:
> >
> > > In core-site.xml, on first, you miss port in the end for HDFS:
> > >
> > > <property>
> > > <name>fs.default.name</name>
> > > <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
> > > </property>
> > >
> > > Regards,
> > > Dejo
> > >
> > > On Wed, Sep 28, 2011 at 6:21 PM, Vinod Gupta Tankala
> > > <tv...@readypulse.com>wrote:
> > >
> > > > Hi,
> > > > I am trying to setup a test system to host a distributed hbase
> > > > installation.
> > > > No matter what I do, I get the below errors.
> > > >
> > > > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Error
> > > > Recovery fo
> > > > r block null bad datanode[0] nodes == null
> > > > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Could
> > not
> > > > get
> > > > blo
> > > > ck locations. Source file "/tmp/mapred/system/jobtracker.info" -
> > > > Aborting...
> > > > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
> > Writing
> > > > to
> > > > fil
> > > > e hdfs://
> > > >
> ec2-184-73-22-146.compute-1.amazonaws.com/tmp/mapred/system/jobtracker
> > .
> > > > info failed!
> > > > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
> > > > FileSystem
> > > > is
> > > > not ready yet!
> > > > 2011-09-28 22:17:26,292 WARN org.apache.hadoop.mapred.JobTracker:
> > Failed
> > > to
> > > > init
> > > > ialize recovery manager.
> > > > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > > > /tmp/mapred/sys
> > > > tem/jobtracker.info could only be replicated to 0 nodes, instead of
> 1
> > > > at
> > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc
> > > > k(FSNamesystem.java:1417)
> > > > ....
> > > >
> > > > this is how i setup my config -
> > > > core-site.xml -
> > > > <configuration>
> > > >
> > > > <property>
> > > > <name>fs.default.name</name>
> > > > <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
> > > > </property>
> > > >
> > > > </configuration>
> > > >
> > > > hdfs-site.xml -
> > > > <configuration>
> > > >
> > > > <property>
> > > > <name>dfs.replication</name>
> > > > <value>1</value>
> > > > </property>
> > > >
> > > > <property>
> > > > <name>dfs.name.dir</name>
> > > > <value>/tmp/hbase</value>
> > > > </property>
> > > >
> > > > <property>
> > > > <name>dfs.data.dir</name>
> > > > <value>/tmp/hbase</value>
> > > > </property>
> > > >
> > > > </configuration>
> > > >
> > > >
> > > > mapred-site.xml -
> > > > <configuration>
> > > >
> > > > <property>
> > > > <name>mapred.job.tracker</name>
> > > > <value>ec2-184-73-22-146.compute-1.amazonaws.com:9001</value>
> > > > </property>
> > > >
> > > > <property>
> > > > <name>mapred.local.dir</name>
> > > > <value>/tmp/mapred_tmp</value>
> > > > </property>
> > > >
> > > > <property>
> > > > <name>mapred.map.tasks</name>
> > > > <value>10</value>
> > > > </property>
> > > >
> > > > <property>
> > > > <name>mapred.reduce.tasks</name>
> > > > <value>2</value>
> > > > </property>
> > > >
> > > > <property>
> > > > <name>mapred.system.dir</name>
> > > > <value>/tmp/mapred/system/</value>
> > > > </property>
> > > >
> > > >
> > > > </configuration>
> > > >
> > > > i know that i am missing something really basic but not sure what it
> > is.
> > > > the
> > > > documentation says mapred.system.dir should be globally accessible.
> how
> > > do
> > > > i
> > > > achieve that?
> > > >
> > > > thanks
> > > > vinod
> > > >
> > >
> >
>
Re: how to set mapred.system.dir?
Posted by Dejan Menges <de...@gmail.com>.
Hi Vinod,
This sounds to me like a DNS issue. I have the same thing on development
environments, and couple of things that made me big headache were:
- Inconsistency of DNS and machine hostname, if not used localhost
- On some systems in /etc/hosts you have localhost defined for both IPv4 and
IPv6 (MacOS is one example) where you need to comment IPv6.
- When I don't want to use localhost at all, I comment it out totally in
/etc/hosts
So, the approach I have in this case is to, prior starting doing anything
with HBase, stop everything, clean the logs, format namenode, and start
Hadoop only. Monitor logs, what won't be problem in this case as that's
stand alone installation, to see if HDFS get up OK. You can also try coping
file from and to your HDFS to be sure HDFS is working fine. Then you can
start MapReduce (JobTracker and TaskTrackers) and later on HBase.
I hope this will help :)
Regards,
Dejo
On Thu, Sep 29, 2011 at 8:06 PM, Vinod Gupta Tankala
<tv...@readypulse.com>wrote:
> Thanks Dejo for pointing that out. I realized that earlier and fixed it.
> But
> I still hit the same problem.
>
> In my case, I only have a single host for now. But I am still trying to do
> a
> distributed setup by listing the machine itself as a slave in config and
> not
> using localhost anywhere. Does this even work? if not, I can try spending
> more time on pseudo-distributed setup for now.
>
> thanks
>
>
> On Thu, Sep 29, 2011 at 4:48 AM, Dejan Menges <dejan.menges@gmail.com
> >wrote:
>
> > In core-site.xml, on first, you miss port in the end for HDFS:
> >
> > <property>
> > <name>fs.default.name</name>
> > <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
> > </property>
> >
> > Regards,
> > Dejo
> >
> > On Wed, Sep 28, 2011 at 6:21 PM, Vinod Gupta Tankala
> > <tv...@readypulse.com>wrote:
> >
> > > Hi,
> > > I am trying to setup a test system to host a distributed hbase
> > > installation.
> > > No matter what I do, I get the below errors.
> > >
> > > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Error
> > > Recovery fo
> > > r block null bad datanode[0] nodes == null
> > > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Could
> not
> > > get
> > > blo
> > > ck locations. Source file "/tmp/mapred/system/jobtracker.info" -
> > > Aborting...
> > > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
> Writing
> > > to
> > > fil
> > > e hdfs://
> > > ec2-184-73-22-146.compute-1.amazonaws.com/tmp/mapred/system/jobtracker
> .
> > > info failed!
> > > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
> > > FileSystem
> > > is
> > > not ready yet!
> > > 2011-09-28 22:17:26,292 WARN org.apache.hadoop.mapred.JobTracker:
> Failed
> > to
> > > init
> > > ialize recovery manager.
> > > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > > /tmp/mapred/sys
> > > tem/jobtracker.info could only be replicated to 0 nodes, instead of 1
> > > at
> > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc
> > > k(FSNamesystem.java:1417)
> > > ....
> > >
> > > this is how i setup my config -
> > > core-site.xml -
> > > <configuration>
> > >
> > > <property>
> > > <name>fs.default.name</name>
> > > <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
> > > </property>
> > >
> > > </configuration>
> > >
> > > hdfs-site.xml -
> > > <configuration>
> > >
> > > <property>
> > > <name>dfs.replication</name>
> > > <value>1</value>
> > > </property>
> > >
> > > <property>
> > > <name>dfs.name.dir</name>
> > > <value>/tmp/hbase</value>
> > > </property>
> > >
> > > <property>
> > > <name>dfs.data.dir</name>
> > > <value>/tmp/hbase</value>
> > > </property>
> > >
> > > </configuration>
> > >
> > >
> > > mapred-site.xml -
> > > <configuration>
> > >
> > > <property>
> > > <name>mapred.job.tracker</name>
> > > <value>ec2-184-73-22-146.compute-1.amazonaws.com:9001</value>
> > > </property>
> > >
> > > <property>
> > > <name>mapred.local.dir</name>
> > > <value>/tmp/mapred_tmp</value>
> > > </property>
> > >
> > > <property>
> > > <name>mapred.map.tasks</name>
> > > <value>10</value>
> > > </property>
> > >
> > > <property>
> > > <name>mapred.reduce.tasks</name>
> > > <value>2</value>
> > > </property>
> > >
> > > <property>
> > > <name>mapred.system.dir</name>
> > > <value>/tmp/mapred/system/</value>
> > > </property>
> > >
> > >
> > > </configuration>
> > >
> > > i know that i am missing something really basic but not sure what it
> is.
> > > the
> > > documentation says mapred.system.dir should be globally accessible. how
> > do
> > > i
> > > achieve that?
> > >
> > > thanks
> > > vinod
> > >
> >
>
Re: how to set mapred.system.dir?
Posted by Vinod Gupta Tankala <tv...@readypulse.com>.
Thanks Dejo for pointing that out. I realized that earlier and fixed it. But
I still hit the same problem.
In my case, I only have a single host for now. But I am still trying to do a
distributed setup by listing the machine itself as a slave in config and not
using localhost anywhere. Does this even work? if not, I can try spending
more time on pseudo-distributed setup for now.
thanks
On Thu, Sep 29, 2011 at 4:48 AM, Dejan Menges <de...@gmail.com>wrote:
> In core-site.xml, on first, you miss port in the end for HDFS:
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
> </property>
>
> Regards,
> Dejo
>
> On Wed, Sep 28, 2011 at 6:21 PM, Vinod Gupta Tankala
> <tv...@readypulse.com>wrote:
>
> > Hi,
> > I am trying to setup a test system to host a distributed hbase
> > installation.
> > No matter what I do, I get the below errors.
> >
> > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Error
> > Recovery fo
> > r block null bad datanode[0] nodes == null
> > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Could not
> > get
> > blo
> > ck locations. Source file "/tmp/mapred/system/jobtracker.info" -
> > Aborting...
> > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker: Writing
> > to
> > fil
> > e hdfs://
> > ec2-184-73-22-146.compute-1.amazonaws.com/tmp/mapred/system/jobtracker.
> > info failed!
> > 2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
> > FileSystem
> > is
> > not ready yet!
> > 2011-09-28 22:17:26,292 WARN org.apache.hadoop.mapred.JobTracker: Failed
> to
> > init
> > ialize recovery manager.
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /tmp/mapred/sys
> > tem/jobtracker.info could only be replicated to 0 nodes, instead of 1
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc
> > k(FSNamesystem.java:1417)
> > ....
> >
> > this is how i setup my config -
> > core-site.xml -
> > <configuration>
> >
> > <property>
> > <name>fs.default.name</name>
> > <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
> > </property>
> >
> > </configuration>
> >
> > hdfs-site.xml -
> > <configuration>
> >
> > <property>
> > <name>dfs.replication</name>
> > <value>1</value>
> > </property>
> >
> > <property>
> > <name>dfs.name.dir</name>
> > <value>/tmp/hbase</value>
> > </property>
> >
> > <property>
> > <name>dfs.data.dir</name>
> > <value>/tmp/hbase</value>
> > </property>
> >
> > </configuration>
> >
> >
> > mapred-site.xml -
> > <configuration>
> >
> > <property>
> > <name>mapred.job.tracker</name>
> > <value>ec2-184-73-22-146.compute-1.amazonaws.com:9001</value>
> > </property>
> >
> > <property>
> > <name>mapred.local.dir</name>
> > <value>/tmp/mapred_tmp</value>
> > </property>
> >
> > <property>
> > <name>mapred.map.tasks</name>
> > <value>10</value>
> > </property>
> >
> > <property>
> > <name>mapred.reduce.tasks</name>
> > <value>2</value>
> > </property>
> >
> > <property>
> > <name>mapred.system.dir</name>
> > <value>/tmp/mapred/system/</value>
> > </property>
> >
> >
> > </configuration>
> >
> > i know that i am missing something really basic but not sure what it is.
> > the
> > documentation says mapred.system.dir should be globally accessible. how
> do
> > i
> > achieve that?
> >
> > thanks
> > vinod
> >
>
Re: how to set mapred.system.dir?
Posted by Dejan Menges <de...@gmail.com>.
In core-site.xml, on first, you miss port in the end for HDFS:
<property>
<name>fs.default.name</name>
<value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
</property>
Regards,
Dejo
On Wed, Sep 28, 2011 at 6:21 PM, Vinod Gupta Tankala
<tv...@readypulse.com>wrote:
> Hi,
> I am trying to setup a test system to host a distributed hbase
> installation.
> No matter what I do, I get the below errors.
>
> 2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery fo
> r block null bad datanode[0] nodes == null
> 2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Could not
> get
> blo
> ck locations. Source file "/tmp/mapred/system/jobtracker.info" -
> Aborting...
> 2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker: Writing
> to
> fil
> e hdfs://
> ec2-184-73-22-146.compute-1.amazonaws.com/tmp/mapred/system/jobtracker.
> info failed!
> 2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
> FileSystem
> is
> not ready yet!
> 2011-09-28 22:17:26,292 WARN org.apache.hadoop.mapred.JobTracker: Failed to
> init
> ialize recovery manager.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /tmp/mapred/sys
> tem/jobtracker.info could only be replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc
> k(FSNamesystem.java:1417)
> ....
>
> this is how i setup my config -
> core-site.xml -
> <configuration>
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
> </property>
>
> </configuration>
>
> hdfs-site.xml -
> <configuration>
>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
>
> <property>
> <name>dfs.name.dir</name>
> <value>/tmp/hbase</value>
> </property>
>
> <property>
> <name>dfs.data.dir</name>
> <value>/tmp/hbase</value>
> </property>
>
> </configuration>
>
>
> mapred-site.xml -
> <configuration>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>ec2-184-73-22-146.compute-1.amazonaws.com:9001</value>
> </property>
>
> <property>
> <name>mapred.local.dir</name>
> <value>/tmp/mapred_tmp</value>
> </property>
>
> <property>
> <name>mapred.map.tasks</name>
> <value>10</value>
> </property>
>
> <property>
> <name>mapred.reduce.tasks</name>
> <value>2</value>
> </property>
>
> <property>
> <name>mapred.system.dir</name>
> <value>/tmp/mapred/system/</value>
> </property>
>
>
> </configuration>
>
> i know that i am missing something really basic but not sure what it is.
> the
> documentation says mapred.system.dir should be globally accessible. how do
> i
> achieve that?
>
> thanks
> vinod
>