You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by howard chen <ho...@gmail.com> on 2006/11/04 15:15:24 UTC
Help in setting Hadoop on multiple servers
Hi
Currently I have 3 servers, A, B, C
1.
I unpacked Hadoop separately on three machines on the same folder (local):
/home/hadoop/
2.
I follow the documentation, set up the JAVA_HOME path, and created a
config folder, on a NFS mounted drive, move the hadoop-env.sh,
hadoop-site.xml & slaves to this folder
/data-0/hadoop_conf/
3.
in the hadoop_conf/slaves, i remove the localhost, but add the 3 server's IP
i.e.
serverA
serverB
serverC
4.
When I type (on serverA): ./start-all.sh --config /data-0/hadoop_conf/
It prompt me to enter password for server A, B, C, but when I typed a
password, I got welcome message from serverA, but I have no way to
enter password for B & C, console stopped here...what can I do?
Thanks.
Re: Help in setting Hadoop on multiple servers
Posted by howard chen <ho...@gmail.com>.
On 11/10/06, Milind Bhandarkar <mi...@yahoo-inc.com> wrote:
>
> The namenode warnings could be a result of one of the two scenarios:
> first, you may have started multiple datanodes on a single machine,
> and therefore the number of machines in DFS and number of datanodes
> are not in sync. This problem is also fixed recently in a patch to
> Hadoop-382. Second, the datanodes do not have enough available
> diskspace to store a block. In any case, the namenode warnings you
> mentioned should not result in the exception you are seeing. You can
> check using "bin/hadoop dfs -ls" command to see if the input
> directory for the map (/tmp/wcin) really exists.
>
> - Milind
>
> On Nov 9, 2006, at 8:37 AM, howard chen wrote:
>
> >
> > Hello,
> >
> > I followed your instruction, now the namenode can be started, good!
> >
> > but when i invoke the example,
> >
> > e.g.
> >
> > bin/hadoop --config ... jar hadoop-0.8.0-examples.jar wordcount -m 1
> > -r 1 /tmp/wcin/ /tmp/wcout/
> >
> > exceptions thrown:
> >
> > java.io.IOException: Input directory /tmp/wcin in server01:50000 is
> > invalid.
> > at org.apache.hadoop.mapred.JobClient.submitJob
> > (JobClient.java:311)
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:
> > 368)
> > at org.apache.hadoop.examples.WordCount.main(WordCount.java:
> > 143)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke
> > (NativeMethodAccessorImpl.java:39)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke
> > (DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at org.apache.hadoop.util.ProgramDriver
> > $ProgramDescription.invoke(ProgramDriver.java:71)
> > at org.apache.hadoop.util.ProgramDriver.driver
> > (ProgramDriver.java:143)
> > at org.apache.hadoop.examples.ExampleDriver.main
> > (ExampleDriver.java:41)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke
> > (NativeMethodAccessorImpl.java:39)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke
> > (DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
> >
> > and from the namenode log, i found many...
> >
> > 2006-11-10 00:17:57,110 WARN org.apache.hadoop.fs.FSNamesystem: Zero
> > targets found, forbidden1.size=4 forbidden2.size()=0
> > 2006-11-10 00:17:57,110 WARN org.apache.hadoop.fs.FSNamesystem: Zero
> > targets found, forbidden1.size=4 forbidden2.size()=0
> > 2006-11-10 00:17:57,110 WARN org.apache.hadoop.fs.FSNamesystem: Zero
> > targets found, forbidden1.size=4 forbidden2.size()=0
> > ....
> >
> > thanks first.
> >
> >
>
>
1. I am using hadoop-0.8.0.tar.gz (DL from hadoop site, not svn)
2. I tried some commands to put files in the DFS, e.g.
$ hadoop --config.... dfs -mkdir /tmp/test-mkdir
$ hadoop -config.... dfs -copyFromLocal ./foo.txt /tmp/test-mkdir/foo.txt
$ hadoop -config.... dfs -put ./some_directory
/tmp/test-mkdir/foo.txt/some_directory
but when i use hadoop --config... dfs -ls
it returns
06/11/10 12:02:13 INFO ipc.Client:
org.apache.hadoop.io.ObjectWritableConnection culler maxidletime=
1000ms
06/11/10 12:02:13 INFO ipc.Client: org.apache.hadoop.io.ObjectWritable
Connection Culler: starting
Found 0 items
3. when reporting, e.g. dfs -report, it returns
Total effective bytes: 1641 (1.60 k)
Effective replication multiplier: 2715720.889092017
-------------------------------------------------
Datanodes available: 4
Name: server4:50010
Total raw bytes: 37843353600 (35.24 GB)
Used raw bytes: 1045558989 (997.12 MB)
% used: 2.76%
Last contact: Fri Nov 10 12:03:06 HKT 2006
Name: server2:50010
Total raw bytes: 37843353600 (35.24 GB)
Used raw bytes: 1087923591 (1.01 GB)
% used: 2.87%
Last contact: Fri Nov 10 12:03:08 HKT 2006
Name: server3:50010
Total raw bytes: 37843353600 (35.24 GB)
Used raw bytes: 1087903520 (1.01 GB)
% used: 2.87%
Last contact: Fri Nov 10 12:03:09 HKT 2006
Name: server1:50010
Total raw bytes: 37843353600 (35.24 GB)
Used raw bytes: 1235111879 (1.15 GB)
% used: 3.26%
Last contact: Fri Nov 10 12:03:09 HKT 2006
Thanks for any comments!
Re: Help in setting Hadoop on multiple servers
Posted by Milind Bhandarkar <mi...@yahoo-inc.com>.
The namenode warnings could be a result of one of the two scenarios:
first, you may have started multiple datanodes on a single machine,
and therefore the number of machines in DFS and number of datanodes
are not in sync. This problem is also fixed recently in a patch to
Hadoop-382. Second, the datanodes do not have enough available
diskspace to store a block. In any case, the namenode warnings you
mentioned should not result in the exception you are seeing. You can
check using "bin/hadoop dfs -ls" command to see if the input
directory for the map (/tmp/wcin) really exists.
- Milind
On Nov 9, 2006, at 8:37 AM, howard chen wrote:
>
> Hello,
>
> I followed your instruction, now the namenode can be started, good!
>
> but when i invoke the example,
>
> e.g.
>
> bin/hadoop --config ... jar hadoop-0.8.0-examples.jar wordcount -m 1
> -r 1 /tmp/wcin/ /tmp/wcout/
>
> exceptions thrown:
>
> java.io.IOException: Input directory /tmp/wcin in server01:50000 is
> invalid.
> at org.apache.hadoop.mapred.JobClient.submitJob
> (JobClient.java:311)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:
> 368)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:
> 143)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.util.ProgramDriver
> $ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.driver
> (ProgramDriver.java:143)
> at org.apache.hadoop.examples.ExampleDriver.main
> (ExampleDriver.java:41)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
>
> and from the namenode log, i found many...
>
> 2006-11-10 00:17:57,110 WARN org.apache.hadoop.fs.FSNamesystem: Zero
> targets found, forbidden1.size=4 forbidden2.size()=0
> 2006-11-10 00:17:57,110 WARN org.apache.hadoop.fs.FSNamesystem: Zero
> targets found, forbidden1.size=4 forbidden2.size()=0
> 2006-11-10 00:17:57,110 WARN org.apache.hadoop.fs.FSNamesystem: Zero
> targets found, forbidden1.size=4 forbidden2.size()=0
> ....
>
> thanks first.
>
>
Re: Help in setting Hadoop on multiple servers
Posted by howard chen <ho...@gmail.com>.
On 11/9/06, Doug Cutting <cu...@apache.org> wrote:
> howard chen wrote:
> > java.io.IOException: NameNode not formatted:
> > /home/projects/tmp/hadoop-web/dfs/name/image
> > at org.apache.hadoop.dfs.FSImage.<init>(FSImage.java:60)
> > at org.apache.hadoop.dfs.FSDirectory.<init>(FSDirectory.java:316)
> > at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:229)
> > at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:142)
> > at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:134)
> > at org.apache.hadoop.dfs.NameNode.main(NameNode.java:582)
> >
> > any more hints? thanks...
>
> Try creating that directory before you format:
>
> mkdir /home/projects/tmp/hadoop-web/dfs/name/image
> bin/hadoop namenode -format
>
> This was a bug that was recently fixed.
>
> http://issues.apache.org/jira/browse/HADOOP-682
>
> Doug
>
Hello,
I followed your instruction, now the namenode can be started, good!
but when i invoke the example,
e.g.
bin/hadoop --config ... jar hadoop-0.8.0-examples.jar wordcount -m 1
-r 1 /tmp/wcin/ /tmp/wcout/
exceptions thrown:
java.io.IOException: Input directory /tmp/wcin in server01:50000 is invalid.
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:311)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:368)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
and from the namenode log, i found many...
2006-11-10 00:17:57,110 WARN org.apache.hadoop.fs.FSNamesystem: Zero
targets found, forbidden1.size=4 forbidden2.size()=0
2006-11-10 00:17:57,110 WARN org.apache.hadoop.fs.FSNamesystem: Zero
targets found, forbidden1.size=4 forbidden2.size()=0
2006-11-10 00:17:57,110 WARN org.apache.hadoop.fs.FSNamesystem: Zero
targets found, forbidden1.size=4 forbidden2.size()=0
....
thanks first.
Re: Help in setting Hadoop on multiple servers
Posted by Doug Cutting <cu...@apache.org>.
howard chen wrote:
> java.io.IOException: NameNode not formatted:
> /home/projects/tmp/hadoop-web/dfs/name/image
> at org.apache.hadoop.dfs.FSImage.<init>(FSImage.java:60)
> at org.apache.hadoop.dfs.FSDirectory.<init>(FSDirectory.java:316)
> at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:229)
> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:142)
> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:134)
> at org.apache.hadoop.dfs.NameNode.main(NameNode.java:582)
>
> any more hints? thanks...
Try creating that directory before you format:
mkdir /home/projects/tmp/hadoop-web/dfs/name/image
bin/hadoop namenode -format
This was a bug that was recently fixed.
http://issues.apache.org/jira/browse/HADOOP-682
Doug
Re: Help in setting Hadoop on multiple servers
Posted by howard chen <ho...@gmail.com>.
On 11/9/06, Lee <le...@gmail.com> wrote:
> Make the --conf argument is before the 'namenode -format' argument. I
> forgot to do this a couple times and it does not ready your config
> otherwise.
>
> ./hadoop --config /your/dir namenode -format
>
> On 11/8/06, howard chen <ho...@gmail.com> wrote:
> >
> > On 11/8/06, Doug Cutting <cu...@apache.org> wrote:
> > > howard chen wrote:
> > > > 2006-11-07 21:53:35,492 ERROR org.apache.hadoop.mapred.TaskTracker:
> > > > Can not start task tracker because java.lang.RuntimeException: Bad
> > > > mapred.job.tracker: local
> > >
> > > To run distributed, you must configure mapred.job.tracker and
> > > fs.default.name to be a host:port pairs on all hosts, typically in the
> > > conf/hadoop-site.xml file. On your slave nodes the values for these are
> > > "local", the default. This should be overridden in hadoop-site.xml.
> > >
> > > Doug
> > >
> >
> > Thanks! It really solved most of the problem...now only one problem
> > left (seems to be...)
> >
> > when I start the server, and stop it, it shows...
> >
> > ./stop-all.sh --config /home_7305/group02/project/hadoop-conf
> >
> >
> > >> no namenode to stop
> >
> > while i look for the namenode log on serverA, it shows:
> >
> > ERROR org.apache.hadoop.dfs.NameNode: java.io.IOException: NameNode
> > not formatted: /hadoop-web/dfs/name/image
> > at org.apache.hadoop.dfs.FSImage.<init>(FSImage.java:60)
> > at org.apache.hadoop.dfs.FSDirectory.<init>(FSDirectory.java:316)
> > at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java
> > :229)
> > at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:142)
> > at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:134)
> > at org.apache.hadoop.dfs.NameNode.main(NameNode.java:582)
> >
> > but i have tried many time to format it using: ./hadoop namenode -format
> >
> > (seems the format only read hadoop-default.xml, but i am using
> > --config switch, and the hadoop.tmp.dir is relocated, is this a
> > problem? but even i removed the hadoop.tmp.dir and use the default
> > /tmp folder, error still occur )
> >
> > thanks again!
> >
>
>
i have tried this, e.g.
hadoop --config .... namenode -format
the command return:
06/11/09 22:34:55 INFO conf.Configuration: parsing
jar:file:/home/projects/bin/hadoop-0.8.0/hadoop-0.8.0.jar!/hadoop-default.xml
but according to the namenode log when running the start-all script
2006-11-09 22:35:27,608 ERROR org.apache.hadoop.dfs.NameNode:
java.io.IOException: NameNode not formatted:
/home/projects/tmp/hadoop-web/dfs/name/image
at org.apache.hadoop.dfs.FSImage.<init>(FSImage.java:60)
at org.apache.hadoop.dfs.FSDirectory.<init>(FSDirectory.java:316)
at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:229)
at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:142)
at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:134)
at org.apache.hadoop.dfs.NameNode.main(NameNode.java:582)
any more hints? thanks...
Re: Help in setting Hadoop on multiple servers
Posted by Lee <le...@gmail.com>.
Make the --conf argument is before the 'namenode -format' argument. I
forgot to do this a couple times and it does not ready your config
otherwise.
./hadoop --config /your/dir namenode -format
On 11/8/06, howard chen <ho...@gmail.com> wrote:
>
> On 11/8/06, Doug Cutting <cu...@apache.org> wrote:
> > howard chen wrote:
> > > 2006-11-07 21:53:35,492 ERROR org.apache.hadoop.mapred.TaskTracker:
> > > Can not start task tracker because java.lang.RuntimeException: Bad
> > > mapred.job.tracker: local
> >
> > To run distributed, you must configure mapred.job.tracker and
> > fs.default.name to be a host:port pairs on all hosts, typically in the
> > conf/hadoop-site.xml file. On your slave nodes the values for these are
> > "local", the default. This should be overridden in hadoop-site.xml.
> >
> > Doug
> >
>
> Thanks! It really solved most of the problem...now only one problem
> left (seems to be...)
>
> when I start the server, and stop it, it shows...
>
> ./stop-all.sh --config /home_7305/group02/project/hadoop-conf
>
>
> >> no namenode to stop
>
> while i look for the namenode log on serverA, it shows:
>
> ERROR org.apache.hadoop.dfs.NameNode: java.io.IOException: NameNode
> not formatted: /hadoop-web/dfs/name/image
> at org.apache.hadoop.dfs.FSImage.<init>(FSImage.java:60)
> at org.apache.hadoop.dfs.FSDirectory.<init>(FSDirectory.java:316)
> at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java
> :229)
> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:142)
> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:134)
> at org.apache.hadoop.dfs.NameNode.main(NameNode.java:582)
>
> but i have tried many time to format it using: ./hadoop namenode -format
>
> (seems the format only read hadoop-default.xml, but i am using
> --config switch, and the hadoop.tmp.dir is relocated, is this a
> problem? but even i removed the hadoop.tmp.dir and use the default
> /tmp folder, error still occur )
>
> thanks again!
>
Re: Help in setting Hadoop on multiple servers
Posted by howard chen <ho...@gmail.com>.
On 11/8/06, Doug Cutting <cu...@apache.org> wrote:
> howard chen wrote:
> > 2006-11-07 21:53:35,492 ERROR org.apache.hadoop.mapred.TaskTracker:
> > Can not start task tracker because java.lang.RuntimeException: Bad
> > mapred.job.tracker: local
>
> To run distributed, you must configure mapred.job.tracker and
> fs.default.name to be a host:port pairs on all hosts, typically in the
> conf/hadoop-site.xml file. On your slave nodes the values for these are
> "local", the default. This should be overridden in hadoop-site.xml.
>
> Doug
>
Thanks! It really solved most of the problem...now only one problem
left (seems to be...)
when I start the server, and stop it, it shows...
./stop-all.sh --config /home_7305/group02/project/hadoop-conf
>> no namenode to stop
while i look for the namenode log on serverA, it shows:
ERROR org.apache.hadoop.dfs.NameNode: java.io.IOException: NameNode
not formatted: /hadoop-web/dfs/name/image
at org.apache.hadoop.dfs.FSImage.<init>(FSImage.java:60)
at org.apache.hadoop.dfs.FSDirectory.<init>(FSDirectory.java:316)
at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:229)
at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:142)
at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:134)
at org.apache.hadoop.dfs.NameNode.main(NameNode.java:582)
but i have tried many time to format it using: ./hadoop namenode -format
(seems the format only read hadoop-default.xml, but i am using
--config switch, and the hadoop.tmp.dir is relocated, is this a
problem? but even i removed the hadoop.tmp.dir and use the default
/tmp folder, error still occur )
thanks again!
Re: Help in setting Hadoop on multiple servers
Posted by Doug Cutting <cu...@apache.org>.
howard chen wrote:
> 2006-11-07 21:53:35,492 ERROR org.apache.hadoop.mapred.TaskTracker:
> Can not start task tracker because java.lang.RuntimeException: Bad
> mapred.job.tracker: local
To run distributed, you must configure mapred.job.tracker and
fs.default.name to be a host:port pairs on all hosts, typically in the
conf/hadoop-site.xml file. On your slave nodes the values for these are
"local", the default. This should be overridden in hadoop-site.xml.
Doug
Re: Help in setting Hadoop on multiple servers
Posted by howard chen <ho...@gmail.com>.
On 11/7/06, Doug Cutting <cu...@apache.org> wrote:
> howard chen wrote:
> > but when I stop-all --config...it show...
> >
> > no jobtracker to stop
> > serverA: Login Success!
> > serverB: Login Success!
> > serverB: no tasktracker to stop
>
> It looks like the tasktracker crashed on startup. Login to ServerB and
> look in its logs to see what happened.
>
> Doug
>
thanks for your reply first.
i tried to look at serverB (startall invoked from serverA)...
tasktracker.log
================
2006-11-07 21:53:35,344 INFO org.apache.hadoop.conf.Configuration:
parsing jar:file:/bin/hadoop-0.8.0/hadoop-0.8.0.jar!/hadoop-default.xml
2006-11-07 21:53:35,448 INFO org.apache.hadoop.conf.Configuration:
parsing jar:file:/bin/hadoop-0.8.0/hadoop-0.8.0.jar!/mapred-default.xml
2006-11-07 21:53:35,492 ERROR org.apache.hadoop.mapred.TaskTracker:
Can not start task tracker because java.lang.RuntimeException: Bad
mapred.job.tracker: local
at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:541)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:384)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1374)
datanode.log
=============
2006-11-07 21:53:32,385 INFO org.apache.hadoop.conf.Configuration:
parsing jar:file:/bin/hadoop-0.8.0/hadoop-0.8.0.jar!/hadoop-default.xml
2006-11-07 21:53:32,527 ERROR org.apache.hadoop.dfs.DataNode:
java.lang.RuntimeException: Not a host:port pair: local
at org.apache.hadoop.dfs.DataNode.createSocketAddr(DataNode.java:84)
at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:158)
at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:1093)
at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1027)
at org.apache.hadoop.dfs.DataNode.runAndWait(DataNode.java:1056)
at org.apache.hadoop.dfs.DataNode.main(DataNode.java:1110)
any hints for debugging?
thanks...
Re: Help in setting Hadoop on multiple servers
Posted by Doug Cutting <cu...@apache.org>.
howard chen wrote:
> but when I stop-all --config...it show...
>
> no jobtracker to stop
> serverA: Login Success!
> serverB: Login Success!
> serverB: no tasktracker to stop
It looks like the tasktracker crashed on startup. Login to ServerB and
look in its logs to see what happened.
Doug
Re: Help in setting Hadoop on multiple servers
Posted by howard chen <ho...@gmail.com>.
On 11/5/06, Andrzej Bialecki <ab...@getopt.org> wrote:
> howard chen wrote:
> > On 11/4/06, Lee <le...@gmail.com> wrote:
> >> You need passwordless ssh setup for the username you start the script
> >> with.
> >>
> >> Lee
> >>
> >> On 11/4/06, howard chen <ho...@gmail.com> wrote:
> >> >
> >> > Hi
> >> >
> >> > Currently I have 3 servers, A, B, C
> >> >
> >> > 1.
> >> >
> >> > I unpacked Hadoop separately on three machines on the same folder
> >> (local):
> >> >
> >> > /home/hadoop/
> >> >
> >> > 2.
> >> >
> >> > I follow the documentation, set up the JAVA_HOME path, and created a
> >> > config folder, on a NFS mounted drive, move the hadoop-env.sh,
> >> > hadoop-site.xml & slaves to this folder
> >> >
> >> > /data-0/hadoop_conf/
> >> >
> >> > 3.
> >> >
> >> > in the hadoop_conf/slaves, i remove the localhost, but add the 3
> >> server's
> >> > IP
> >> >
> >> > i.e.
> >> > serverA
> >> > serverB
> >> > serverC
> >> >
> >> >
> >> > 4.
> >> >
> >> > When I type (on serverA): ./start-all.sh --config /data-0/hadoop_conf/
> >> >
> >> > It prompt me to enter password for server A, B, C, but when I typed a
> >> > password, I got welcome message from serverA, but I have no way to
> >> > enter password for B & C, console stopped here...what can I do?
> >> >
> >> > Thanks.
> >> >
> >>
> >>
> >
> > if I my system don't allow passwordless ssh, are there any way
> > workaround?
>
> "Passwordless" doesn't mean that the account in question needs to have
> no password. You can use public/private key authentication, i.e. add the
> master's public key to .ssh/authorized_keys file on each node.
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
>
Thanks! this solved the login problem when starting the client!
1.
Now I can successfully start-all from serverA using...
./start-all.sh --config ...
starting namenode, logging to...
serverA: Login Success!
serverC: starting datanode, logging to...
serverB: Login Success!
serverA: starting datanode, logging to...
serverB: Login Success!
serverB: starting datanode, logging to...
serverD: Login Success!
serverD: starting datanode, logging to...
starting jobtracker, logging to ...
serverA: Login Success!
serverC: Login Success!
serverC: starting tasktracker, logging to...
serverA: starting tasktracker, logging to...
serverB: Login Success!
serverB: starting tasktracker, logging to...
serverD: Login Success!
serverD: starting tasktracker, logging to...
but when I stop-all --config...it show...
no jobtracker to stop
serverA: Login Success!
serverB: Login Success!
serverB: no tasktracker to stop
serverC: Login Success!
serverA: no tasktracker to stop
serverC: no tasktracker to stop
serverD: Login Success!
serverD: no tasktracker to stop
no namenode to stop
serverA: Login Success!
serverC: Login Success!
serverC: no datanode to stop
serverD: Login Success!
serverA: no datanode to stop
serverD: no datanode to stop
serverB: Login Success!
serverB: no datanode to stop
Is that something wrong?
2. I can run the example-word count on a single machine, but how to
verify the example is running on serveral machine?
Thanks...
Re: Help in setting Hadoop on multiple servers
Posted by Andrzej Bialecki <ab...@getopt.org>.
howard chen wrote:
> On 11/4/06, Lee <le...@gmail.com> wrote:
>> You need passwordless ssh setup for the username you start the script
>> with.
>>
>> Lee
>>
>> On 11/4/06, howard chen <ho...@gmail.com> wrote:
>> >
>> > Hi
>> >
>> > Currently I have 3 servers, A, B, C
>> >
>> > 1.
>> >
>> > I unpacked Hadoop separately on three machines on the same folder
>> (local):
>> >
>> > /home/hadoop/
>> >
>> > 2.
>> >
>> > I follow the documentation, set up the JAVA_HOME path, and created a
>> > config folder, on a NFS mounted drive, move the hadoop-env.sh,
>> > hadoop-site.xml & slaves to this folder
>> >
>> > /data-0/hadoop_conf/
>> >
>> > 3.
>> >
>> > in the hadoop_conf/slaves, i remove the localhost, but add the 3
>> server's
>> > IP
>> >
>> > i.e.
>> > serverA
>> > serverB
>> > serverC
>> >
>> >
>> > 4.
>> >
>> > When I type (on serverA): ./start-all.sh --config /data-0/hadoop_conf/
>> >
>> > It prompt me to enter password for server A, B, C, but when I typed a
>> > password, I got welcome message from serverA, but I have no way to
>> > enter password for B & C, console stopped here...what can I do?
>> >
>> > Thanks.
>> >
>>
>>
>
> if I my system don't allow passwordless ssh, are there any way
> workaround?
"Passwordless" doesn't mean that the account in question needs to have
no password. You can use public/private key authentication, i.e. add the
master's public key to .ssh/authorized_keys file on each node.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Help in setting Hadoop on multiple servers
Posted by Lee <le...@gmail.com>.
Not sure if it will work, but you could try starting the daemon script
locally on each box in your slaves file.
On 11/4/06, howard chen <ho...@gmail.com> wrote:
>
> On 11/4/06, Lee <le...@gmail.com> wrote:
> > You need passwordless ssh setup for the username you start the script
> with.
> >
> > Lee
> >
> > On 11/4/06, howard chen <ho...@gmail.com> wrote:
> > >
> > > Hi
> > >
> > > Currently I have 3 servers, A, B, C
> > >
> > > 1.
> > >
> > > I unpacked Hadoop separately on three machines on the same folder
> (local):
> > >
> > > /home/hadoop/
> > >
> > > 2.
> > >
> > > I follow the documentation, set up the JAVA_HOME path, and created a
> > > config folder, on a NFS mounted drive, move the hadoop-env.sh,
> > > hadoop-site.xml & slaves to this folder
> > >
> > > /data-0/hadoop_conf/
> > >
> > > 3.
> > >
> > > in the hadoop_conf/slaves, i remove the localhost, but add the 3
> server's
> > > IP
> > >
> > > i.e.
> > > serverA
> > > serverB
> > > serverC
> > >
> > >
> > > 4.
> > >
> > > When I type (on serverA): ./start-all.sh --config /data-0/hadoop_conf/
> > >
> > > It prompt me to enter password for server A, B, C, but when I typed a
> > > password, I got welcome message from serverA, but I have no way to
> > > enter password for B & C, console stopped here...what can I do?
> > >
> > > Thanks.
> > >
> >
> >
>
> if I my system don't allow passwordless ssh, are there any way workaround?
>
> thanks
>
Re: Help in setting Hadoop on multiple servers
Posted by howard chen <ho...@gmail.com>.
On 11/4/06, Lee <le...@gmail.com> wrote:
> You need passwordless ssh setup for the username you start the script with.
>
> Lee
>
> On 11/4/06, howard chen <ho...@gmail.com> wrote:
> >
> > Hi
> >
> > Currently I have 3 servers, A, B, C
> >
> > 1.
> >
> > I unpacked Hadoop separately on three machines on the same folder (local):
> >
> > /home/hadoop/
> >
> > 2.
> >
> > I follow the documentation, set up the JAVA_HOME path, and created a
> > config folder, on a NFS mounted drive, move the hadoop-env.sh,
> > hadoop-site.xml & slaves to this folder
> >
> > /data-0/hadoop_conf/
> >
> > 3.
> >
> > in the hadoop_conf/slaves, i remove the localhost, but add the 3 server's
> > IP
> >
> > i.e.
> > serverA
> > serverB
> > serverC
> >
> >
> > 4.
> >
> > When I type (on serverA): ./start-all.sh --config /data-0/hadoop_conf/
> >
> > It prompt me to enter password for server A, B, C, but when I typed a
> > password, I got welcome message from serverA, but I have no way to
> > enter password for B & C, console stopped here...what can I do?
> >
> > Thanks.
> >
>
>
if I my system don't allow passwordless ssh, are there any way workaround?
thanks
Re: Help in setting Hadoop on multiple servers
Posted by Lee <le...@gmail.com>.
You need passwordless ssh setup for the username you start the script with.
Lee
On 11/4/06, howard chen <ho...@gmail.com> wrote:
>
> Hi
>
> Currently I have 3 servers, A, B, C
>
> 1.
>
> I unpacked Hadoop separately on three machines on the same folder (local):
>
> /home/hadoop/
>
> 2.
>
> I follow the documentation, set up the JAVA_HOME path, and created a
> config folder, on a NFS mounted drive, move the hadoop-env.sh,
> hadoop-site.xml & slaves to this folder
>
> /data-0/hadoop_conf/
>
> 3.
>
> in the hadoop_conf/slaves, i remove the localhost, but add the 3 server's
> IP
>
> i.e.
> serverA
> serverB
> serverC
>
>
> 4.
>
> When I type (on serverA): ./start-all.sh --config /data-0/hadoop_conf/
>
> It prompt me to enter password for server A, B, C, but when I typed a
> password, I got welcome message from serverA, but I have no way to
> enter password for B & C, console stopped here...what can I do?
>
> Thanks.
>