You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mithila Nagendra <mn...@asu.edu> on 2008/11/19 19:12:17 UTC
Hadoop Installation
Hello
I m currently a student at Arizona State University, Tempe, Arizona,
pursuing my masters in Computer Science. I m currently involved in a
research project that makes use of Hadoop to run various map reduce
functions. Hence I searched the web on whats the best way to install Hadoop
on different nodes in a cluster, and I stumbled upon your website.
I used your tutorial on "How to install hadoop on a linux system" by Michael
Noll to setup Hadoop on a UNIX system. I have a few questions related to it:
1. Does it matter that I m installing hadoop on UNIX and not on LINUX - do I
have to follow different steps?
2. The configuration for hadoop-site.xml - does it remain the same no matter
what platform is being used? Do I just type the same thing out in the file
hadoop-site.xml present in the hadoop installation on the node?
3. When I try to start the daemons by executing the command
conf/start-all.sh, I get an exception which says "hadoop: user specified log
class 'org.apache.commons.logging.impl.Log4JLogger' cannot be found or is
not usable" - this happens when tasktracker is being started. What steps do
I take to deal with this exception?
I could send you the screen shot of the exception if you wish. It would be
of immense help if you could provide answers for the above questions.
Thank you! Looking forward to your reply.
Best Regards
Mithila Nagendra
Fwd: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
Hey guys!!
Any ideas on this one! I m still stuck with this!
I tried dropping the jar files into the lib. It still doesnt work.. The
following is how the lib looks after the new files were put in:
[mithila@node01 hadoop-0.17.2.1]$ cd bin
[mithila@node01 bin]$ ls
hadoop hadoop-daemon.sh rcc start-all.sh
start-dfs.sh stop-all.sh stop-dfs.sh
hadoop-config.sh hadoop-daemons.sh slaves.sh start-balancer.sh
start-mapred.sh stop-balancer.sh stop-mapred.sh
[mithila@node01 bin]$ cd ..
[mithila@node01 hadoop-0.17.2.1]$ mv commons-logging-1.1.1/* lib
[mithila@node01 hadoop-0.17.2.1]$ cd lib
[mithila@node01 lib]$ ls
commons-cli-2.0-SNAPSHOT.jar commons-logging-1.1.1-javadoc.jar
commons-logging-tests.jar junit-3.8.1.jar log4j-1.2.13.jar site
commons-codec-1.3.jar commons-logging-1.1.1-sources.jar
jets3t-0.5.0.jar junit-3.8.1.LICENSE.txt native
xmlenc-0.52.jar
commons-httpclient-3.0.1.jar commons-logging-adapters-1.1.1.jar
jetty-5.1.4.jar kfs-0.1.jar NOTICE.txt
commons-logging-1.0.4.jar commons-logging-api-1.0.4.jar
jetty-5.1.4.LICENSE.txt kfs-0.1.LICENSE.txt RELEASE-NOTES.txt
commons-logging-1.1.1.jar commons-logging-api-1.1.1.jar jetty-ext
LICENSE.txt servlet-api.jar
[mithila@node01 lib]$ cd ..
[mithila@node01 hadoop-0.17.2.1]$ bin/start-all.sh
starting namenode, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-namenode-node01.out
mithila@localhost's password:
localhost: starting datanode, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-datanode-node01.out
mithila@localhost's password:
localhost: starting secondarynamenode, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-secondarynamenode-node01.out
starting jobtracker, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-jobtracker-node01.out
mithila@localhost's password:
localhost: starting tasktracker, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-tasktracker-node01.out
localhost: Exception in thread "main" java.lang.ExceptionInInitializerError
localhost: Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
cannot be found or is not useable.
localhost: at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
localhost: at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
localhost: at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
localhost: at
org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
localhost: at
org.apache.hadoop.mapred.TaskTracker.<clinit>(TaskTracker.java:95)
localhost: Could not find the main class:
org.apache.hadoop.mapred.TaskTracker. Program will exit.
[mithila@node01 hadoop-0.17.2.1]$ ls
bin c++ commons-logging-1.1.1 contrib
hadoop-0.17.2.1-core.jar hadoop-0.17.2.1-test.jar libhdfs logs
README.txt webapps
build.xml CHANGES.txt conf docs
hadoop-0.17.2.1-examples.jar lib LICENSE.txt
NOTICE.txt src
[mithila@node01 hadoop-0.17.2.1]$ conf/hadoop namenode -format
bash: conf/hadoop: No such file or directory
[mithila@node01 hadoop-0.17.2.1]$ bin/hadoop namenode -format
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
cannot be found or is not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
at org.apache.hadoop.dfs.NameNode.<clinit>(NameNode.java:88)
Could not find the main class: org.apache.hadoop.dfs.NameNode. Program will
exit.
[mithila@node01 hadoop-0.17.2.1]$
Mithila
On Fri, Nov 21, 2008 at 9:22 PM, Alex Loddengaard <al...@cloudera.com> wrote:
> Download the 1.1.1.tar.gz binaries. This file will have a bunch of JAR
> files; drop the JAR files in to $HADOOP_HOME/lib and see what happens.
> Alex
>
> On Fri, Nov 21, 2008 at 9:19 AM, Mithila Nagendra <mn...@asu.edu>
> wrote:
>
> > Hey ALex
> > Which file do I download from the apache commons website?
> >
> > Thanks
> > Mithila
> > On Fri, Nov 21, 2008 at 8:15 PM, Mithila Nagendra <mn...@asu.edu>
> > wrote:
> >
> > > I tried the 0.18.2 as welll.. it gave me the same exception.. so tried
> > the
> > > lower version.. I should check if this works.. Thanks!
> > >
> > >
> > > On Fri, Nov 21, 2008 at 5:06 AM, Alex Loddengaard <alex@cloudera.com
> > >wrote:
> > >
> > >> Maybe try downloading the Apache Commons - Logging jars (<
> > >> http://commons.apache.org/downloads/download_logging.cgi>) and drop
> > them
> > >> in
> > >> to $HADOOP_HOME/lib.
> > >> Just curious, if you're starting a new cluster, why have you chosen to
> > use
> > >> 0.17.* and not 0.18.2? It would be a good idea to use 0.18.2 if
> > possible.
> > >>
> > >> Alex
> > >>
> > >> On Thu, Nov 20, 2008 at 4:36 PM, Mithila Nagendra <mn...@asu.edu>
> > >> wrote:
> > >>
> > >> > Hey
> > >> > The version is: Linux enpc3740.eas.asu.edu 2.6.9-67.0.20.EL #1 Wed
> > Jun
> > >> 18
> > >> > 12:23:46 EDT 2008 i686 i686 i386 GNU/Linux, this is what I got when
> I
> > >> used
> > >> > the command uname -a (thanks Tom!)
> > >> >
> > >> > Yea it is bin/start-all.. Following is the exception that I got when
> i
> > >> > tried
> > >> > to start the daemons..
> > >> >
> > >> >
> > >> > [mithila@node01 mithila]$ ls
> > >> > hadoop-0.17.2.1 hadoop-0.18.2 hadoop-0.18.2.tar.gz
> > >> > [mithila@node01 mithila]$ cd hadoop-0.17*
> > >> > [mithila@node01 hadoop-0.17.2.1]$ ls
> > >> > bin c++ conf docs
> > >> > hadoop-0.17.2.1-examples.jar lib LICENSE.txt NOTICE.txt src
> > >> > build.xml CHANGES.txt contrib hadoop-0.17.2.1-core.jar
> > >> > hadoop-0.17.2.1-test.jar libhdfs logs README.txt
> > webapps
> > >> > [mithila@node01 hadoop-0.17.2.1]$ bin/start-all
> > >> > bash: bin/start-all: No such file or directory
> > >> > [mithila@node01 hadoop-0.17.2.1]$ bin/start-all.sh
> > >> > starting namenode, logging to
> > >> >
> > >> >
> > >>
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-namenode-node01.out
> > >> > mithila@localhost's password:
> > >> > localhost: starting datanode, logging to
> > >> >
> > >> >
> > >>
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-datanode-node01.out
> > >> > mithila@localhost's password:
> > >> > localhost: starting secondarynamenode, logging to
> > >> >
> > >> >
> > >>
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-secondarynamenode-node01.out
> > >> > starting jobtracker, logging to
> > >> >
> > >> >
> > >>
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-jobtracker-node01.out
> > >> > mithila@localhost's password:
> > >> > localhost: starting tasktracker, logging to
> > >> >
> > >> >
> > >>
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-tasktracker-node01.out
> > >> > localhost: Exception in thread "main"
> > >> java.lang.ExceptionInInitializerError
> > >> > localhost: Caused by:
> > >> org.apache.commons.logging.LogConfigurationException:
> > >> > User-specified log class
> 'org.apache.commons.logging.impl.Log4JLogger'
> > >> > cannot be found or is not useable.
> > >> > localhost: at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
> > >> > localhost: at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
> > >> > localhost: at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
> > >> > localhost: at
> > >> > org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
> > >> > localhost: at
> > >> > org.apache.hadoop.mapred.TaskTracker.<clinit>(TaskTracker.java:95)
> > >> > localhost: Could not find the main class:
> > >> > org.apache.hadoop.mapred.TaskTracker. Program will exit.
> > >> >
> > >> > AND when I tried formatting the file system I got the following
> > >> exception..
> > >> > I followed Michael Noll s step to install Hadoop.. I m currently
> > working
> > >> on
> > >> > a single node and if this works will move on to multiple nodes in a
> > >> > cluster.
> > >> >
> > >> > [mithila@node01 hadoop-0.17.2.1]$ bin/hadoop namenode -format
> > >> > Exception in thread "main" java.lang.ExceptionInInitializerError
> > >> > Caused by: org.apache.commons.logging.LogConfigurationException:
> > >> > User-specified log class
> 'org.apache.commons.logging.impl.Log4JLogger'
> > >> > cannot be found or is not useable.
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
> > >> > at
> > >> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
> > >> > at org.apache.hadoop.dfs.NameNode.<clinit>(NameNode.java:88)
> > >> > Could not find the main class: org.apache.hadoop.dfs.NameNode.
> > Program
> > >> > will
> > >> > exit.
> > >> >
> > >> >
> > >> > I have no idea whats wrong... my hadoop-xml file looks as follows:
> > >> >
> > >> > <?xml version="1.0"?>
> > >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> > >> >
> > >> > <!-- Put site-specific property overrides in this file. -->
> > >> >
> > >> > <configuration>
> > >> >
> > >> > <property>
> > >> > <name>hadoop.tmp.dir</name>
> > >> > <value>/tmp/hadoop-${user.name}</value>
> > >> > <description>A base for other temporary directories</description>
> > >> > </property>
> > >> >
> > >> >
> > >> > <property>
> > >> > <name>fs.default.name</name>
> > >> > <value>hdfs://localhost:54310</value>
> > >> > <description>The name of the default file system. A URI whose
> > >> > scheme and authority determine the FileSystem implementation. The
> > >> > URI's scheme determines the config property (fs.scheme.impl) naming
> > >> > the FileSystem implementation class. The URI's authority is used to
> > >> > determine the host, port, etc for a filesystem.</description>
> > >> > </property>
> > >> >
> > >> >
> > >> > <property>
> > >> > <name>mapred.job.tracker</name>
> > >> > <value>localhost:54311</value>
> > >> > <description>The host and port that the MapReduce job tracker runs
> at.
> > >> > If "local", then jobs are run in-process as a single map and
> > >> > reduce task.</description>
> > >> > </property>
> > >> >
> > >> >
> > >> > <property>
> > >> > <name>dfs.replication</name>
> > >> > <value>1</value>
> > >> > <description>Default block replication.
> > >> > The actual number of replications can be specified when the file is
> > >> > created.
> > >> > The default is used if replication is not specified in create
> > >> > time.</description>
> > >> > </property>
> > >> > "conf/hadoop-site.xml" 42L, 1271C
> > >> >
> > >> >
> > >> > My hadoop-env.sh looks as follows:
> > >> >
> > >> > # Set Hadoop-specific environment variables here.
> > >> >
> > >> > # The only required environment variable is JAVA_HOME. All others
> are
> > >> > # optional. When running a distributed configuration it is best to
> > >> > # set JAVA_HOME in this file, so that it is correctly defined on
> > >> > # remote nodes.
> > >> >
> > >> > # The java implementation to use. Required.
> > >> > export JAVA_HOME=/usr/java/jdk1.6.0_10
> > >> >
> > >> > # Extra Java CLASSPATH elements. Optional.
> > >> > # export HADOOP_CLASSPATH=
> > >> >
> > >> > # The maximum amount of heap to use, in MB. Default is 1000.
> > >> > # export HADOOP_HEAPSIZE=2000
> > >> >
> > >> > # Extra Java runtime options. Empty by default.
> > >> > # export HADOOP_OPTS=-server
> > >> >
> > >> > # Command specific options appended to HADOOP_OPTS when specified
> > >> > export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> > >> > $HADOOP_NAMENODE_OPTS"
> > >> > export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
> > >> > $HADOOP_SECONDARYNAMENODE_OPTS"
> > >> > export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
> > >> > $HADOOP_DATANODE_OPTS"
> > >> > export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
> > >> > $HADOOP_BALANCER_OPTS"
> > >> > export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
> > >> > $HADOOP_JOBTRACKER_OPTS"
> > >> > # export HADOOP_TASKTRACKER_OPTS=
> > >> > # The following applies to multiple commands (fs, dfs, fsck, distcp
> > etc)
> > >> > # export HADOOP_CLIENT_OPTS
> > >> >
> > >> > # Extra ssh options. Empty by default.
> > >> > # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o
> > >> SendEnv=HADOOP_CONF_DIR"
> > >> >
> > >> > # Where log files are stored. $HADOOP_HOME/logs by default.
> > >> > # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
> > >> >
> > >> > # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by
> > default.
> > >> > # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
> > >> >
> > >> > # host:path where hadoop code should be rsync'd from. Unset by
> > default.
> > >> > # export HADOOP_MASTER=master:/home/$USER/src/hadoop
> > >> >
> > >> > "conf/hadoop-env.sh" 54L, 2236C
> > >> >
> > >> > Dont know what the exceptions mean.. Does anyone have an idea?
> > >> >
> > >> > THanks
> > >> > Mithila
> > >> >
> > >> >
> > >> > On Thu, Nov 20, 2008 at 6:42 AM, some speed <sp...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > I am working on the same for my master's project and i know how
> > >> > frustrating
> > >> > > it can be to get hadoop installed.
> > >> > > If time is not a factor, I suggest you first try implementing it
> in
> > a
> > >> > > psuedo distributed environment. Once you understand how things
> work
> > by
> > >> > > implementing a simple map reduce program, u can easily move on to
> a
> > >> > cluster.
> > >> > >
> > >> > > From what little i know, let me tell u a few things,
> > >> > >
> > >> > > I tried using the university network to install hadoop.. it was a
> > real
> > >> > > pain. mayb it was coz I didnt have the admin privileges( to
> install
> > >> HDFS
> > >> > n
> > >> > > its files). so make sure u have admin rights or u keep getting an
> > >> error
> > >> > of
> > >> > > port 22 (for ssh) being not opened or the demeons were not
> started.
> > >> > > n btw is it conf/start-all.sh?? i think its bin/start -all.sh or
> > >> > something
> > >> > > of that sort.
> > >> > >
> > >> > > hadoop-site.xml -- i had the links bookmarked somewhere- cant
> find
> > it
> > >> > now
> > >> > > but i think u are supposed to have a few more details in there for
> a
> > >> > cluster
> > >> > > installation. Am sure we can find those online quite easily.
> > >> > >
> > >> > > Also i suppose u are using java? if u are good at eclipse, then u
> > can
> > >> > > implement map reduce/hadoop thru that on a single node (just to
> get
> > a
> > >> > hang
> > >> > > of it).
> > >> > >
> > >> > > All the best!
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Wed, Nov 19, 2008 at 6:38 PM, Tom Wheeler <to...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > >> On Wed, Nov 19, 2008 at 5:31 PM, Mithila Nagendra <
> > mnagendr@asu.edu>
> > >> > >> wrote:
> > >> > >> > Oh is that so. Im not sure which UNIX it is since Im working
> with
> > a
> > >> > >> cluster
> > >> > >> > that is remotely accessed.
> > >> > >>
> > >> > >> If you can get a shell on the machine, try typing "uname -a" to
> see
> > >> > >> what type of UNIX it is.
> > >> > >>
> > >> > >> Alternatively, the os.name, os.version and os.arch Java system
> > >> > >> properties could also help you to identify the operating system.
> > >> > >>
> > >> > >> --
> > >> > >> Tom Wheeler
> > >> > >> http://www.tomwheeler.com/
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
Thanks Steve! Will take a look at it..
Mithila
On Mon, Nov 24, 2008 at 6:32 PM, Steve Loughran <st...@apache.org> wrote:
> Mithila Nagendra wrote:
>
>> I tried dropping the jar files into the lib. It still doesnt work.. The
>> following is how the lib looks after the new files were put in:
>>
>> [mithila@node01 hadoop-0.17.2.1]$ cd bin
>> [mithila@node01 bin]$ ls
>> hadoop hadoop-daemon.sh rcc start-all.sh
>> start-dfs.sh stop-all.sh stop-dfs.sh
>> hadoop-config.sh hadoop-daemons.sh slaves.sh start-balancer.sh
>> start-mapred.sh stop-balancer.sh stop-mapred.sh
>> [mithila@node01 bin]$ cd ..
>> [mithila@node01 hadoop-0.17.2.1]$ mv commons-logging-1.1.1/* lib
>> [mithila@node01 hadoop-0.17.2.1]$ cd lib
>> [mithila@node01 lib]$ ls
>> commons-cli-2.0-SNAPSHOT.jar commons-logging-1.1.1-javadoc.jar
>> commons-logging-tests.jar junit-3.8.1.jar log4j-1.2.13.jar
>> site
>> commons-codec-1.3.jar commons-logging-1.1.1-sources.jar
>> jets3t-0.5.0.jar junit-3.8.1.LICENSE.txt native
>> xmlenc-0.52.jar
>> commons-httpclient-3.0.1.jar commons-logging-adapters-1.1.1.jar
>> jetty-5.1.4.jar kfs-0.1.jar NOTICE.txt
>> commons-logging-1.0.4.jar commons-logging-api-1.0.4.jar
>> jetty-5.1.4.LICENSE.txt kfs-0.1.LICENSE.txt RELEASE-NOTES.txt
>> commons-logging-1.1.1.jar commons-logging-api-1.1.1.jar
>> jetty-ext
>> LICENSE.txt servlet-api.jar
>>
>
> OK, you now have two copies of commons-logging in there. I would detele the
> -1.0.4 version and the -api and -sources JARs.
>
> But I don't think that is the root cause of this problem. Are you running
> on a linux system that has commons-logging installed as an RPM or .deb
> package? Because that could be making a real mess of your classpath. The
> error your are seeing implies that log4j isnt there or it won't load. And as
> log4j is there, it looks like a classloader problem of some sort. These are
> tractable, but they are hard to track down and it is only on your system(s)
> that it exists. There's not much that can be done remotely.
>
> -steve
>
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
Hello all
The exception was being throw cause the JDK version I had was a lower one.
The minute I updated that it worked well! So I guess having the right
version of java matters alot!!
Thanks for your help!
Mithila
On Tue, Nov 25, 2008 at 9:57 AM, Steve Loughran <st...@apache.org> wrote:
> Mithila Nagendra wrote:
>
>> Hey steve
>> The version is: Linux enpc3740.eas.asu.edu 2.6.9-67.0.20.EL #1 Wed Jun 18
>> 12:23:46 EDT 2008 i686 i686 i386 GNU/Linux, this is what I got when I used
>> the command uname -a
>>
>> On Tue, Nov 25, 2008 at 1:50 PM, Steve Loughran <st...@apache.org>
>> wrote:
>>
>> Mithila Nagendra wrote:
>>>
>>> Hey Steve
>>>> I deleted what ever I needed to.. still no luck..
>>>>
>>>> You said that the classpath might be messed up.. Is there some way I can
>>>> reset it? For the root user? What path do I set it to.
>>>>
>>>>
>>>> Let's start with what kind of machine is this? Windows? or Linux. If
>>> Linux,
>>> which one?
>>>
>>>
>>
> OK
>
> 1. In yum (redhat) or the synaptic package manager, is there any package
> called log4j installed? or liblog4j?
> 2. Install ant, and run
> ant -diagnostics
> email us the results
>
Re: Hadoop Installation
Posted by Steve Loughran <st...@apache.org>.
Mithila Nagendra wrote:
> Hey steve
> The version is: Linux enpc3740.eas.asu.edu 2.6.9-67.0.20.EL #1 Wed Jun 18
> 12:23:46 EDT 2008 i686 i686 i386 GNU/Linux, this is what I got when I used
> the command uname -a
>
> On Tue, Nov 25, 2008 at 1:50 PM, Steve Loughran <st...@apache.org> wrote:
>
>> Mithila Nagendra wrote:
>>
>>> Hey Steve
>>> I deleted what ever I needed to.. still no luck..
>>>
>>> You said that the classpath might be messed up.. Is there some way I can
>>> reset it? For the root user? What path do I set it to.
>>>
>>>
>> Let's start with what kind of machine is this? Windows? or Linux. If Linux,
>> which one?
>>
>
OK
1. In yum (redhat) or the synaptic package manager, is there any package
called log4j installed? or liblog4j?
2. Install ant, and run
ant -diagnostics
email us the results
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
Hey steve
The version is: Linux enpc3740.eas.asu.edu 2.6.9-67.0.20.EL #1 Wed Jun 18
12:23:46 EDT 2008 i686 i686 i386 GNU/Linux, this is what I got when I used
the command uname -a
On Tue, Nov 25, 2008 at 1:50 PM, Steve Loughran <st...@apache.org> wrote:
> Mithila Nagendra wrote:
>
>> Hey Steve
>> I deleted what ever I needed to.. still no luck..
>>
>> You said that the classpath might be messed up.. Is there some way I can
>> reset it? For the root user? What path do I set it to.
>>
>>
> Let's start with what kind of machine is this? Windows? or Linux. If Linux,
> which one?
>
Re: Hadoop Installation
Posted by Steve Loughran <st...@apache.org>.
Mithila Nagendra wrote:
> Hey Steve
> I deleted what ever I needed to.. still no luck..
>
> You said that the classpath might be messed up.. Is there some way I can
> reset it? For the root user? What path do I set it to.
>
Let's start with what kind of machine is this? Windows? or Linux. If
Linux, which one?
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
Hey Steve
I deleted what ever I needed to.. still no luck..
You said that the classpath might be messed up.. Is there some way I can
reset it? For the root user? What path do I set it to.
Mithila
On Mon, Nov 24, 2008 at 8:54 PM, Steve Loughran <st...@apache.org> wrote:
> Mithila Nagendra wrote:
>
>> Hey Steve
>>
>> Out of the following which one do I remove - just making sure.. I got rid
>> of commons-logging-1.0.4.jar
>>
>> commons-logging-api-1.0.4.jar
>> commons-logging-1.1.1-sources.jar commons-logging-1.1.1-sources.jar
>>
>
> Hadoop is currently built with commons-logging-1.0.4.jar, so strictly
> speaking that should be the only one to retain; all the others can go. But
> you could also delete everythiing except commons-logging-1.1.1.jar and it
> should work just as well. None of the -sources JARs are needed, none of the
> -api jars are needed.
> -steve
>
Re: Hadoop Installation
Posted by Steve Loughran <st...@apache.org>.
Mithila Nagendra wrote:
> Hey Steve
>
> Out of the following which one do I remove - just making sure.. I got rid
> of commons-logging-1.0.4.jar
>
> commons-logging-api-1.0.4.jar
> commons-logging-1.1.1-sources.jar commons-logging-1.1.1-sources.jar
Hadoop is currently built with commons-logging-1.0.4.jar, so strictly
speaking that should be the only one to retain; all the others can go.
But you could also delete everythiing except commons-logging-1.1.1.jar
and it should work just as well. None of the -sources JARs are needed,
none of the -api jars are needed.
-steve
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
Hey Steve
Out of the following which one do I remove - just making sure.. I got rid
of commons-logging-1.0.4.jar
commons-logging-api-1.0.4.jar
commons-logging-1.1.1-sources.jar commons-logging-1.1.1-sources.jar
Thanks!
Mithila
On Mon, Nov 24, 2008 at 6:32 PM, Steve Loughran <st...@apache.org> wrote:
> Mithila Nagendra wrote:
>
>> I tried dropping the jar files into the lib. It still doesnt work.. The
>> following is how the lib looks after the new files were put in:
>>
>> [mithila@node01 hadoop-0.17.2.1]$ cd bin
>> [mithila@node01 bin]$ ls
>> hadoop hadoop-daemon.sh rcc start-all.sh
>> start-dfs.sh stop-all.sh stop-dfs.sh
>> hadoop-config.sh hadoop-daemons.sh slaves.sh start-balancer.sh
>> start-mapred.sh stop-balancer.sh stop-mapred.sh
>> [mithila@node01 bin]$ cd ..
>> [mithila@node01 hadoop-0.17.2.1]$ mv commons-logging-1.1.1/* lib
>> [mithila@node01 hadoop-0.17.2.1]$ cd lib
>> [mithila@node01 lib]$ ls
>> commons-cli-2.0-SNAPSHOT.jar commons-logging-1.1.1-javadoc.jar
>> commons-logging-tests.jar junit-3.8.1.jar log4j-1.2.13.jar
>> site
>> commons-codec-1.3.jar commons-logging-1.1.1-sources.jar
>> jets3t-0.5.0.jar junit-3.8.1.LICENSE.txt native
>> xmlenc-0.52.jar
>> commons-httpclient-3.0.1.jar commons-logging-adapters-1.1.1.jar
>> jetty-5.1.4.jar kfs-0.1.jar NOTICE.txt
>> commons-logging-1.0.4.jar commons-logging-api-1.0.4.jar
>> jetty-5.1.4.LICENSE.txt kfs-0.1.LICENSE.txt RELEASE-NOTES.txt
>> commons-logging-1.1.1.jar commons-logging-api-1.1.1.jar
>> jetty-ext
>> LICENSE.txt servlet-api.jar
>>
>
> OK, you now have two copies of commons-logging in there. I would detele the
> -1.0.4 version and the -api and -sources JARs.
>
> But I don't think that is the root cause of this problem. Are you running
> on a linux system that has commons-logging installed as an RPM or .deb
> package? Because that could be making a real mess of your classpath. The
> error your are seeing implies that log4j isnt there or it won't load. And as
> log4j is there, it looks like a classloader problem of some sort. These are
> tractable, but they are hard to track down and it is only on your system(s)
> that it exists. There's not much that can be done remotely.
>
> -steve
>
Re: Hadoop Installation
Posted by Steve Loughran <st...@apache.org>.
Mithila Nagendra wrote:
> I tried dropping the jar files into the lib. It still doesnt work.. The
> following is how the lib looks after the new files were put in:
>
> [mithila@node01 hadoop-0.17.2.1]$ cd bin
> [mithila@node01 bin]$ ls
> hadoop hadoop-daemon.sh rcc start-all.sh
> start-dfs.sh stop-all.sh stop-dfs.sh
> hadoop-config.sh hadoop-daemons.sh slaves.sh start-balancer.sh
> start-mapred.sh stop-balancer.sh stop-mapred.sh
> [mithila@node01 bin]$ cd ..
> [mithila@node01 hadoop-0.17.2.1]$ mv commons-logging-1.1.1/* lib
> [mithila@node01 hadoop-0.17.2.1]$ cd lib
> [mithila@node01 lib]$ ls
> commons-cli-2.0-SNAPSHOT.jar commons-logging-1.1.1-javadoc.jar
> commons-logging-tests.jar junit-3.8.1.jar log4j-1.2.13.jar site
> commons-codec-1.3.jar commons-logging-1.1.1-sources.jar
> jets3t-0.5.0.jar junit-3.8.1.LICENSE.txt native
> xmlenc-0.52.jar
> commons-httpclient-3.0.1.jar commons-logging-adapters-1.1.1.jar
> jetty-5.1.4.jar kfs-0.1.jar NOTICE.txt
> commons-logging-1.0.4.jar commons-logging-api-1.0.4.jar
> jetty-5.1.4.LICENSE.txt kfs-0.1.LICENSE.txt RELEASE-NOTES.txt
> commons-logging-1.1.1.jar commons-logging-api-1.1.1.jar jetty-ext
> LICENSE.txt servlet-api.jar
OK, you now have two copies of commons-logging in there. I would detele
the -1.0.4 version and the -api and -sources JARs.
But I don't think that is the root cause of this problem. Are you
running on a linux system that has commons-logging installed as an RPM
or .deb package? Because that could be making a real mess of your
classpath. The error your are seeing implies that log4j isnt there or it
won't load. And as log4j is there, it looks like a classloader problem
of some sort. These are tractable, but they are hard to track down and
it is only on your system(s) that it exists. There's not much that can
be done remotely.
-steve
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
I tried dropping the jar files into the lib. It still doesnt work.. The
following is how the lib looks after the new files were put in:
[mithila@node01 hadoop-0.17.2.1]$ cd bin
[mithila@node01 bin]$ ls
hadoop hadoop-daemon.sh rcc start-all.sh
start-dfs.sh stop-all.sh stop-dfs.sh
hadoop-config.sh hadoop-daemons.sh slaves.sh start-balancer.sh
start-mapred.sh stop-balancer.sh stop-mapred.sh
[mithila@node01 bin]$ cd ..
[mithila@node01 hadoop-0.17.2.1]$ mv commons-logging-1.1.1/* lib
[mithila@node01 hadoop-0.17.2.1]$ cd lib
[mithila@node01 lib]$ ls
commons-cli-2.0-SNAPSHOT.jar commons-logging-1.1.1-javadoc.jar
commons-logging-tests.jar junit-3.8.1.jar log4j-1.2.13.jar site
commons-codec-1.3.jar commons-logging-1.1.1-sources.jar
jets3t-0.5.0.jar junit-3.8.1.LICENSE.txt native
xmlenc-0.52.jar
commons-httpclient-3.0.1.jar commons-logging-adapters-1.1.1.jar
jetty-5.1.4.jar kfs-0.1.jar NOTICE.txt
commons-logging-1.0.4.jar commons-logging-api-1.0.4.jar
jetty-5.1.4.LICENSE.txt kfs-0.1.LICENSE.txt RELEASE-NOTES.txt
commons-logging-1.1.1.jar commons-logging-api-1.1.1.jar jetty-ext
LICENSE.txt servlet-api.jar
[mithila@node01 lib]$ cd ..
[mithila@node01 hadoop-0.17.2.1]$ bin/start-all.sh
starting namenode, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-namenode-node01.out
mithila@localhost's password:
localhost: starting datanode, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-datanode-node01.out
mithila@localhost's password:
localhost: starting secondarynamenode, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-secondarynamenode-node01.out
starting jobtracker, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-jobtracker-node01.out
mithila@localhost's password:
localhost: starting tasktracker, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-tasktracker-node01.out
localhost: Exception in thread "main" java.lang.ExceptionInInitializerError
localhost: Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
cannot be found or is not useable.
localhost: at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
localhost: at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
localhost: at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
localhost: at
org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
localhost: at
org.apache.hadoop.mapred.TaskTracker.<clinit>(TaskTracker.java:95)
localhost: Could not find the main class:
org.apache.hadoop.mapred.TaskTracker. Program will exit.
[mithila@node01 hadoop-0.17.2.1]$ ls
bin c++ commons-logging-1.1.1 contrib
hadoop-0.17.2.1-core.jar hadoop-0.17.2.1-test.jar libhdfs logs
README.txt webapps
build.xml CHANGES.txt conf docs
hadoop-0.17.2.1-examples.jar lib LICENSE.txt
NOTICE.txt src
[mithila@node01 hadoop-0.17.2.1]$ conf/hadoop namenode -format
bash: conf/hadoop: No such file or directory
[mithila@node01 hadoop-0.17.2.1]$ bin/hadoop namenode -format
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
cannot be found or is not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
at org.apache.hadoop.dfs.NameNode.<clinit>(NameNode.java:88)
Could not find the main class: org.apache.hadoop.dfs.NameNode. Program will
exit.
[mithila@node01 hadoop-0.17.2.1]$
Mithila
On Fri, Nov 21, 2008 at 9:22 PM, Alex Loddengaard <al...@cloudera.com> wrote:
> Download the 1.1.1.tar.gz binaries. This file will have a bunch of JAR
> files; drop the JAR files in to $HADOOP_HOME/lib and see what happens.
> Alex
>
> On Fri, Nov 21, 2008 at 9:19 AM, Mithila Nagendra <mn...@asu.edu>
> wrote:
>
> > Hey ALex
> > Which file do I download from the apache commons website?
> >
> > Thanks
> > Mithila
> > On Fri, Nov 21, 2008 at 8:15 PM, Mithila Nagendra <mn...@asu.edu>
> > wrote:
> >
> > > I tried the 0.18.2 as welll.. it gave me the same exception.. so tried
> > the
> > > lower version.. I should check if this works.. Thanks!
> > >
> > >
> > > On Fri, Nov 21, 2008 at 5:06 AM, Alex Loddengaard <alex@cloudera.com
> > >wrote:
> > >
> > >> Maybe try downloading the Apache Commons - Logging jars (<
> > >> http://commons.apache.org/downloads/download_logging.cgi>) and drop
> > them
> > >> in
> > >> to $HADOOP_HOME/lib.
> > >> Just curious, if you're starting a new cluster, why have you chosen to
> > use
> > >> 0.17.* and not 0.18.2? It would be a good idea to use 0.18.2 if
> > possible.
> > >>
> > >> Alex
> > >>
> > >> On Thu, Nov 20, 2008 at 4:36 PM, Mithila Nagendra <mn...@asu.edu>
> > >> wrote:
> > >>
> > >> > Hey
> > >> > The version is: Linux enpc3740.eas.asu.edu 2.6.9-67.0.20.EL #1 Wed
> > Jun
> > >> 18
> > >> > 12:23:46 EDT 2008 i686 i686 i386 GNU/Linux, this is what I got when
> I
> > >> used
> > >> > the command uname -a (thanks Tom!)
> > >> >
> > >> > Yea it is bin/start-all.. Following is the exception that I got when
> i
> > >> > tried
> > >> > to start the daemons..
> > >> >
> > >> >
> > >> > [mithila@node01 mithila]$ ls
> > >> > hadoop-0.17.2.1 hadoop-0.18.2 hadoop-0.18.2.tar.gz
> > >> > [mithila@node01 mithila]$ cd hadoop-0.17*
> > >> > [mithila@node01 hadoop-0.17.2.1]$ ls
> > >> > bin c++ conf docs
> > >> > hadoop-0.17.2.1-examples.jar lib LICENSE.txt NOTICE.txt src
> > >> > build.xml CHANGES.txt contrib hadoop-0.17.2.1-core.jar
> > >> > hadoop-0.17.2.1-test.jar libhdfs logs README.txt
> > webapps
> > >> > [mithila@node01 hadoop-0.17.2.1]$ bin/start-all
> > >> > bash: bin/start-all: No such file or directory
> > >> > [mithila@node01 hadoop-0.17.2.1]$ bin/start-all.sh
> > >> > starting namenode, logging to
> > >> >
> > >> >
> > >>
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-namenode-node01.out
> > >> > mithila@localhost's password:
> > >> > localhost: starting datanode, logging to
> > >> >
> > >> >
> > >>
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-datanode-node01.out
> > >> > mithila@localhost's password:
> > >> > localhost: starting secondarynamenode, logging to
> > >> >
> > >> >
> > >>
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-secondarynamenode-node01.out
> > >> > starting jobtracker, logging to
> > >> >
> > >> >
> > >>
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-jobtracker-node01.out
> > >> > mithila@localhost's password:
> > >> > localhost: starting tasktracker, logging to
> > >> >
> > >> >
> > >>
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-tasktracker-node01.out
> > >> > localhost: Exception in thread "main"
> > >> java.lang.ExceptionInInitializerError
> > >> > localhost: Caused by:
> > >> org.apache.commons.logging.LogConfigurationException:
> > >> > User-specified log class
> 'org.apache.commons.logging.impl.Log4JLogger'
> > >> > cannot be found or is not useable.
> > >> > localhost: at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
> > >> > localhost: at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
> > >> > localhost: at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
> > >> > localhost: at
> > >> > org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
> > >> > localhost: at
> > >> > org.apache.hadoop.mapred.TaskTracker.<clinit>(TaskTracker.java:95)
> > >> > localhost: Could not find the main class:
> > >> > org.apache.hadoop.mapred.TaskTracker. Program will exit.
> > >> >
> > >> > AND when I tried formatting the file system I got the following
> > >> exception..
> > >> > I followed Michael Noll s step to install Hadoop.. I m currently
> > working
> > >> on
> > >> > a single node and if this works will move on to multiple nodes in a
> > >> > cluster.
> > >> >
> > >> > [mithila@node01 hadoop-0.17.2.1]$ bin/hadoop namenode -format
> > >> > Exception in thread "main" java.lang.ExceptionInInitializerError
> > >> > Caused by: org.apache.commons.logging.LogConfigurationException:
> > >> > User-specified log class
> 'org.apache.commons.logging.impl.Log4JLogger'
> > >> > cannot be found or is not useable.
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
> > >> > at
> > >> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
> > >> > at org.apache.hadoop.dfs.NameNode.<clinit>(NameNode.java:88)
> > >> > Could not find the main class: org.apache.hadoop.dfs.NameNode.
> > Program
> > >> > will
> > >> > exit.
> > >> >
> > >> >
> > >> > I have no idea whats wrong... my hadoop-xml file looks as follows:
> > >> >
> > >> > <?xml version="1.0"?>
> > >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> > >> >
> > >> > <!-- Put site-specific property overrides in this file. -->
> > >> >
> > >> > <configuration>
> > >> >
> > >> > <property>
> > >> > <name>hadoop.tmp.dir</name>
> > >> > <value>/tmp/hadoop-${user.name}</value>
> > >> > <description>A base for other temporary directories</description>
> > >> > </property>
> > >> >
> > >> >
> > >> > <property>
> > >> > <name>fs.default.name</name>
> > >> > <value>hdfs://localhost:54310</value>
> > >> > <description>The name of the default file system. A URI whose
> > >> > scheme and authority determine the FileSystem implementation. The
> > >> > URI's scheme determines the config property (fs.scheme.impl) naming
> > >> > the FileSystem implementation class. The URI's authority is used to
> > >> > determine the host, port, etc for a filesystem.</description>
> > >> > </property>
> > >> >
> > >> >
> > >> > <property>
> > >> > <name>mapred.job.tracker</name>
> > >> > <value>localhost:54311</value>
> > >> > <description>The host and port that the MapReduce job tracker runs
> at.
> > >> > If "local", then jobs are run in-process as a single map and
> > >> > reduce task.</description>
> > >> > </property>
> > >> >
> > >> >
> > >> > <property>
> > >> > <name>dfs.replication</name>
> > >> > <value>1</value>
> > >> > <description>Default block replication.
> > >> > The actual number of replications can be specified when the file is
> > >> > created.
> > >> > The default is used if replication is not specified in create
> > >> > time.</description>
> > >> > </property>
> > >> > "conf/hadoop-site.xml" 42L, 1271C
> > >> >
> > >> >
> > >> > My hadoop-env.sh looks as follows:
> > >> >
> > >> > # Set Hadoop-specific environment variables here.
> > >> >
> > >> > # The only required environment variable is JAVA_HOME. All others
> are
> > >> > # optional. When running a distributed configuration it is best to
> > >> > # set JAVA_HOME in this file, so that it is correctly defined on
> > >> > # remote nodes.
> > >> >
> > >> > # The java implementation to use. Required.
> > >> > export JAVA_HOME=/usr/java/jdk1.6.0_10
> > >> >
> > >> > # Extra Java CLASSPATH elements. Optional.
> > >> > # export HADOOP_CLASSPATH=
> > >> >
> > >> > # The maximum amount of heap to use, in MB. Default is 1000.
> > >> > # export HADOOP_HEAPSIZE=2000
> > >> >
> > >> > # Extra Java runtime options. Empty by default.
> > >> > # export HADOOP_OPTS=-server
> > >> >
> > >> > # Command specific options appended to HADOOP_OPTS when specified
> > >> > export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> > >> > $HADOOP_NAMENODE_OPTS"
> > >> > export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
> > >> > $HADOOP_SECONDARYNAMENODE_OPTS"
> > >> > export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
> > >> > $HADOOP_DATANODE_OPTS"
> > >> > export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
> > >> > $HADOOP_BALANCER_OPTS"
> > >> > export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
> > >> > $HADOOP_JOBTRACKER_OPTS"
> > >> > # export HADOOP_TASKTRACKER_OPTS=
> > >> > # The following applies to multiple commands (fs, dfs, fsck, distcp
> > etc)
> > >> > # export HADOOP_CLIENT_OPTS
> > >> >
> > >> > # Extra ssh options. Empty by default.
> > >> > # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o
> > >> SendEnv=HADOOP_CONF_DIR"
> > >> >
> > >> > # Where log files are stored. $HADOOP_HOME/logs by default.
> > >> > # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
> > >> >
> > >> > # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by
> > default.
> > >> > # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
> > >> >
> > >> > # host:path where hadoop code should be rsync'd from. Unset by
> > default.
> > >> > # export HADOOP_MASTER=master:/home/$USER/src/hadoop
> > >> >
> > >> > "conf/hadoop-env.sh" 54L, 2236C
> > >> >
> > >> > Dont know what the exceptions mean.. Does anyone have an idea?
> > >> >
> > >> > THanks
> > >> > Mithila
> > >> >
> > >> >
> > >> > On Thu, Nov 20, 2008 at 6:42 AM, some speed <sp...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > I am working on the same for my master's project and i know how
> > >> > frustrating
> > >> > > it can be to get hadoop installed.
> > >> > > If time is not a factor, I suggest you first try implementing it
> in
> > a
> > >> > > psuedo distributed environment. Once you understand how things
> work
> > by
> > >> > > implementing a simple map reduce program, u can easily move on to
> a
> > >> > cluster.
> > >> > >
> > >> > > From what little i know, let me tell u a few things,
> > >> > >
> > >> > > I tried using the university network to install hadoop.. it was a
> > real
> > >> > > pain. mayb it was coz I didnt have the admin privileges( to
> install
> > >> HDFS
> > >> > n
> > >> > > its files). so make sure u have admin rights or u keep getting an
> > >> error
> > >> > of
> > >> > > port 22 (for ssh) being not opened or the demeons were not
> started.
> > >> > > n btw is it conf/start-all.sh?? i think its bin/start -all.sh or
> > >> > something
> > >> > > of that sort.
> > >> > >
> > >> > > hadoop-site.xml -- i had the links bookmarked somewhere- cant
> find
> > it
> > >> > now
> > >> > > but i think u are supposed to have a few more details in there for
> a
> > >> > cluster
> > >> > > installation. Am sure we can find those online quite easily.
> > >> > >
> > >> > > Also i suppose u are using java? if u are good at eclipse, then u
> > can
> > >> > > implement map reduce/hadoop thru that on a single node (just to
> get
> > a
> > >> > hang
> > >> > > of it).
> > >> > >
> > >> > > All the best!
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Wed, Nov 19, 2008 at 6:38 PM, Tom Wheeler <to...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > >> On Wed, Nov 19, 2008 at 5:31 PM, Mithila Nagendra <
> > mnagendr@asu.edu>
> > >> > >> wrote:
> > >> > >> > Oh is that so. Im not sure which UNIX it is since Im working
> with
> > a
> > >> > >> cluster
> > >> > >> > that is remotely accessed.
> > >> > >>
> > >> > >> If you can get a shell on the machine, try typing "uname -a" to
> see
> > >> > >> what type of UNIX it is.
> > >> > >>
> > >> > >> Alternatively, the os.name, os.version and os.arch Java system
> > >> > >> properties could also help you to identify the operating system.
> > >> > >>
> > >> > >> --
> > >> > >> Tom Wheeler
> > >> > >> http://www.tomwheeler.com/
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Hadoop Installation
Posted by Alex Loddengaard <al...@cloudera.com>.
Download the 1.1.1.tar.gz binaries. This file will have a bunch of JAR
files; drop the JAR files in to $HADOOP_HOME/lib and see what happens.
Alex
On Fri, Nov 21, 2008 at 9:19 AM, Mithila Nagendra <mn...@asu.edu> wrote:
> Hey ALex
> Which file do I download from the apache commons website?
>
> Thanks
> Mithila
> On Fri, Nov 21, 2008 at 8:15 PM, Mithila Nagendra <mn...@asu.edu>
> wrote:
>
> > I tried the 0.18.2 as welll.. it gave me the same exception.. so tried
> the
> > lower version.. I should check if this works.. Thanks!
> >
> >
> > On Fri, Nov 21, 2008 at 5:06 AM, Alex Loddengaard <alex@cloudera.com
> >wrote:
> >
> >> Maybe try downloading the Apache Commons - Logging jars (<
> >> http://commons.apache.org/downloads/download_logging.cgi>) and drop
> them
> >> in
> >> to $HADOOP_HOME/lib.
> >> Just curious, if you're starting a new cluster, why have you chosen to
> use
> >> 0.17.* and not 0.18.2? It would be a good idea to use 0.18.2 if
> possible.
> >>
> >> Alex
> >>
> >> On Thu, Nov 20, 2008 at 4:36 PM, Mithila Nagendra <mn...@asu.edu>
> >> wrote:
> >>
> >> > Hey
> >> > The version is: Linux enpc3740.eas.asu.edu 2.6.9-67.0.20.EL #1 Wed
> Jun
> >> 18
> >> > 12:23:46 EDT 2008 i686 i686 i386 GNU/Linux, this is what I got when I
> >> used
> >> > the command uname -a (thanks Tom!)
> >> >
> >> > Yea it is bin/start-all.. Following is the exception that I got when i
> >> > tried
> >> > to start the daemons..
> >> >
> >> >
> >> > [mithila@node01 mithila]$ ls
> >> > hadoop-0.17.2.1 hadoop-0.18.2 hadoop-0.18.2.tar.gz
> >> > [mithila@node01 mithila]$ cd hadoop-0.17*
> >> > [mithila@node01 hadoop-0.17.2.1]$ ls
> >> > bin c++ conf docs
> >> > hadoop-0.17.2.1-examples.jar lib LICENSE.txt NOTICE.txt src
> >> > build.xml CHANGES.txt contrib hadoop-0.17.2.1-core.jar
> >> > hadoop-0.17.2.1-test.jar libhdfs logs README.txt
> webapps
> >> > [mithila@node01 hadoop-0.17.2.1]$ bin/start-all
> >> > bash: bin/start-all: No such file or directory
> >> > [mithila@node01 hadoop-0.17.2.1]$ bin/start-all.sh
> >> > starting namenode, logging to
> >> >
> >> >
> >>
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-namenode-node01.out
> >> > mithila@localhost's password:
> >> > localhost: starting datanode, logging to
> >> >
> >> >
> >>
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-datanode-node01.out
> >> > mithila@localhost's password:
> >> > localhost: starting secondarynamenode, logging to
> >> >
> >> >
> >>
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-secondarynamenode-node01.out
> >> > starting jobtracker, logging to
> >> >
> >> >
> >>
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-jobtracker-node01.out
> >> > mithila@localhost's password:
> >> > localhost: starting tasktracker, logging to
> >> >
> >> >
> >>
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-tasktracker-node01.out
> >> > localhost: Exception in thread "main"
> >> java.lang.ExceptionInInitializerError
> >> > localhost: Caused by:
> >> org.apache.commons.logging.LogConfigurationException:
> >> > User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
> >> > cannot be found or is not useable.
> >> > localhost: at
> >> >
> >> >
> >>
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
> >> > localhost: at
> >> >
> >> >
> >>
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
> >> > localhost: at
> >> >
> >> >
> >>
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
> >> > localhost: at
> >> > org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
> >> > localhost: at
> >> > org.apache.hadoop.mapred.TaskTracker.<clinit>(TaskTracker.java:95)
> >> > localhost: Could not find the main class:
> >> > org.apache.hadoop.mapred.TaskTracker. Program will exit.
> >> >
> >> > AND when I tried formatting the file system I got the following
> >> exception..
> >> > I followed Michael Noll s step to install Hadoop.. I m currently
> working
> >> on
> >> > a single node and if this works will move on to multiple nodes in a
> >> > cluster.
> >> >
> >> > [mithila@node01 hadoop-0.17.2.1]$ bin/hadoop namenode -format
> >> > Exception in thread "main" java.lang.ExceptionInInitializerError
> >> > Caused by: org.apache.commons.logging.LogConfigurationException:
> >> > User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
> >> > cannot be found or is not useable.
> >> > at
> >> >
> >> >
> >>
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
> >> > at
> >> >
> >> >
> >>
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
> >> > at
> >> >
> >> >
> >>
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
> >> > at
> >> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
> >> > at org.apache.hadoop.dfs.NameNode.<clinit>(NameNode.java:88)
> >> > Could not find the main class: org.apache.hadoop.dfs.NameNode.
> Program
> >> > will
> >> > exit.
> >> >
> >> >
> >> > I have no idea whats wrong... my hadoop-xml file looks as follows:
> >> >
> >> > <?xml version="1.0"?>
> >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >> >
> >> > <!-- Put site-specific property overrides in this file. -->
> >> >
> >> > <configuration>
> >> >
> >> > <property>
> >> > <name>hadoop.tmp.dir</name>
> >> > <value>/tmp/hadoop-${user.name}</value>
> >> > <description>A base for other temporary directories</description>
> >> > </property>
> >> >
> >> >
> >> > <property>
> >> > <name>fs.default.name</name>
> >> > <value>hdfs://localhost:54310</value>
> >> > <description>The name of the default file system. A URI whose
> >> > scheme and authority determine the FileSystem implementation. The
> >> > URI's scheme determines the config property (fs.scheme.impl) naming
> >> > the FileSystem implementation class. The URI's authority is used to
> >> > determine the host, port, etc for a filesystem.</description>
> >> > </property>
> >> >
> >> >
> >> > <property>
> >> > <name>mapred.job.tracker</name>
> >> > <value>localhost:54311</value>
> >> > <description>The host and port that the MapReduce job tracker runs at.
> >> > If "local", then jobs are run in-process as a single map and
> >> > reduce task.</description>
> >> > </property>
> >> >
> >> >
> >> > <property>
> >> > <name>dfs.replication</name>
> >> > <value>1</value>
> >> > <description>Default block replication.
> >> > The actual number of replications can be specified when the file is
> >> > created.
> >> > The default is used if replication is not specified in create
> >> > time.</description>
> >> > </property>
> >> > "conf/hadoop-site.xml" 42L, 1271C
> >> >
> >> >
> >> > My hadoop-env.sh looks as follows:
> >> >
> >> > # Set Hadoop-specific environment variables here.
> >> >
> >> > # The only required environment variable is JAVA_HOME. All others are
> >> > # optional. When running a distributed configuration it is best to
> >> > # set JAVA_HOME in this file, so that it is correctly defined on
> >> > # remote nodes.
> >> >
> >> > # The java implementation to use. Required.
> >> > export JAVA_HOME=/usr/java/jdk1.6.0_10
> >> >
> >> > # Extra Java CLASSPATH elements. Optional.
> >> > # export HADOOP_CLASSPATH=
> >> >
> >> > # The maximum amount of heap to use, in MB. Default is 1000.
> >> > # export HADOOP_HEAPSIZE=2000
> >> >
> >> > # Extra Java runtime options. Empty by default.
> >> > # export HADOOP_OPTS=-server
> >> >
> >> > # Command specific options appended to HADOOP_OPTS when specified
> >> > export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> >> > $HADOOP_NAMENODE_OPTS"
> >> > export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
> >> > $HADOOP_SECONDARYNAMENODE_OPTS"
> >> > export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
> >> > $HADOOP_DATANODE_OPTS"
> >> > export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
> >> > $HADOOP_BALANCER_OPTS"
> >> > export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
> >> > $HADOOP_JOBTRACKER_OPTS"
> >> > # export HADOOP_TASKTRACKER_OPTS=
> >> > # The following applies to multiple commands (fs, dfs, fsck, distcp
> etc)
> >> > # export HADOOP_CLIENT_OPTS
> >> >
> >> > # Extra ssh options. Empty by default.
> >> > # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o
> >> SendEnv=HADOOP_CONF_DIR"
> >> >
> >> > # Where log files are stored. $HADOOP_HOME/logs by default.
> >> > # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
> >> >
> >> > # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by
> default.
> >> > # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
> >> >
> >> > # host:path where hadoop code should be rsync'd from. Unset by
> default.
> >> > # export HADOOP_MASTER=master:/home/$USER/src/hadoop
> >> >
> >> > "conf/hadoop-env.sh" 54L, 2236C
> >> >
> >> > Dont know what the exceptions mean.. Does anyone have an idea?
> >> >
> >> > THanks
> >> > Mithila
> >> >
> >> >
> >> > On Thu, Nov 20, 2008 at 6:42 AM, some speed <sp...@gmail.com>
> >> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > I am working on the same for my master's project and i know how
> >> > frustrating
> >> > > it can be to get hadoop installed.
> >> > > If time is not a factor, I suggest you first try implementing it in
> a
> >> > > psuedo distributed environment. Once you understand how things work
> by
> >> > > implementing a simple map reduce program, u can easily move on to a
> >> > cluster.
> >> > >
> >> > > From what little i know, let me tell u a few things,
> >> > >
> >> > > I tried using the university network to install hadoop.. it was a
> real
> >> > > pain. mayb it was coz I didnt have the admin privileges( to install
> >> HDFS
> >> > n
> >> > > its files). so make sure u have admin rights or u keep getting an
> >> error
> >> > of
> >> > > port 22 (for ssh) being not opened or the demeons were not started.
> >> > > n btw is it conf/start-all.sh?? i think its bin/start -all.sh or
> >> > something
> >> > > of that sort.
> >> > >
> >> > > hadoop-site.xml -- i had the links bookmarked somewhere- cant find
> it
> >> > now
> >> > > but i think u are supposed to have a few more details in there for a
> >> > cluster
> >> > > installation. Am sure we can find those online quite easily.
> >> > >
> >> > > Also i suppose u are using java? if u are good at eclipse, then u
> can
> >> > > implement map reduce/hadoop thru that on a single node (just to get
> a
> >> > hang
> >> > > of it).
> >> > >
> >> > > All the best!
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Nov 19, 2008 at 6:38 PM, Tom Wheeler <to...@gmail.com>
> >> wrote:
> >> > >
> >> > >> On Wed, Nov 19, 2008 at 5:31 PM, Mithila Nagendra <
> mnagendr@asu.edu>
> >> > >> wrote:
> >> > >> > Oh is that so. Im not sure which UNIX it is since Im working with
> a
> >> > >> cluster
> >> > >> > that is remotely accessed.
> >> > >>
> >> > >> If you can get a shell on the machine, try typing "uname -a" to see
> >> > >> what type of UNIX it is.
> >> > >>
> >> > >> Alternatively, the os.name, os.version and os.arch Java system
> >> > >> properties could also help you to identify the operating system.
> >> > >>
> >> > >> --
> >> > >> Tom Wheeler
> >> > >> http://www.tomwheeler.com/
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
Hey ALex
Which file do I download from the apache commons website?
Thanks
Mithila
On Fri, Nov 21, 2008 at 8:15 PM, Mithila Nagendra <mn...@asu.edu> wrote:
> I tried the 0.18.2 as welll.. it gave me the same exception.. so tried the
> lower version.. I should check if this works.. Thanks!
>
>
> On Fri, Nov 21, 2008 at 5:06 AM, Alex Loddengaard <al...@cloudera.com>wrote:
>
>> Maybe try downloading the Apache Commons - Logging jars (<
>> http://commons.apache.org/downloads/download_logging.cgi>) and drop them
>> in
>> to $HADOOP_HOME/lib.
>> Just curious, if you're starting a new cluster, why have you chosen to use
>> 0.17.* and not 0.18.2? It would be a good idea to use 0.18.2 if possible.
>>
>> Alex
>>
>> On Thu, Nov 20, 2008 at 4:36 PM, Mithila Nagendra <mn...@asu.edu>
>> wrote:
>>
>> > Hey
>> > The version is: Linux enpc3740.eas.asu.edu 2.6.9-67.0.20.EL #1 Wed Jun
>> 18
>> > 12:23:46 EDT 2008 i686 i686 i386 GNU/Linux, this is what I got when I
>> used
>> > the command uname -a (thanks Tom!)
>> >
>> > Yea it is bin/start-all.. Following is the exception that I got when i
>> > tried
>> > to start the daemons..
>> >
>> >
>> > [mithila@node01 mithila]$ ls
>> > hadoop-0.17.2.1 hadoop-0.18.2 hadoop-0.18.2.tar.gz
>> > [mithila@node01 mithila]$ cd hadoop-0.17*
>> > [mithila@node01 hadoop-0.17.2.1]$ ls
>> > bin c++ conf docs
>> > hadoop-0.17.2.1-examples.jar lib LICENSE.txt NOTICE.txt src
>> > build.xml CHANGES.txt contrib hadoop-0.17.2.1-core.jar
>> > hadoop-0.17.2.1-test.jar libhdfs logs README.txt webapps
>> > [mithila@node01 hadoop-0.17.2.1]$ bin/start-all
>> > bash: bin/start-all: No such file or directory
>> > [mithila@node01 hadoop-0.17.2.1]$ bin/start-all.sh
>> > starting namenode, logging to
>> >
>> >
>> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-namenode-node01.out
>> > mithila@localhost's password:
>> > localhost: starting datanode, logging to
>> >
>> >
>> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-datanode-node01.out
>> > mithila@localhost's password:
>> > localhost: starting secondarynamenode, logging to
>> >
>> >
>> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-secondarynamenode-node01.out
>> > starting jobtracker, logging to
>> >
>> >
>> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-jobtracker-node01.out
>> > mithila@localhost's password:
>> > localhost: starting tasktracker, logging to
>> >
>> >
>> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-tasktracker-node01.out
>> > localhost: Exception in thread "main"
>> java.lang.ExceptionInInitializerError
>> > localhost: Caused by:
>> org.apache.commons.logging.LogConfigurationException:
>> > User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
>> > cannot be found or is not useable.
>> > localhost: at
>> >
>> >
>> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
>> > localhost: at
>> >
>> >
>> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
>> > localhost: at
>> >
>> >
>> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
>> > localhost: at
>> > org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
>> > localhost: at
>> > org.apache.hadoop.mapred.TaskTracker.<clinit>(TaskTracker.java:95)
>> > localhost: Could not find the main class:
>> > org.apache.hadoop.mapred.TaskTracker. Program will exit.
>> >
>> > AND when I tried formatting the file system I got the following
>> exception..
>> > I followed Michael Noll s step to install Hadoop.. I m currently working
>> on
>> > a single node and if this works will move on to multiple nodes in a
>> > cluster.
>> >
>> > [mithila@node01 hadoop-0.17.2.1]$ bin/hadoop namenode -format
>> > Exception in thread "main" java.lang.ExceptionInInitializerError
>> > Caused by: org.apache.commons.logging.LogConfigurationException:
>> > User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
>> > cannot be found or is not useable.
>> > at
>> >
>> >
>> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
>> > at
>> >
>> >
>> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
>> > at
>> >
>> >
>> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
>> > at
>> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
>> > at org.apache.hadoop.dfs.NameNode.<clinit>(NameNode.java:88)
>> > Could not find the main class: org.apache.hadoop.dfs.NameNode. Program
>> > will
>> > exit.
>> >
>> >
>> > I have no idea whats wrong... my hadoop-xml file looks as follows:
>> >
>> > <?xml version="1.0"?>
>> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >
>> > <!-- Put site-specific property overrides in this file. -->
>> >
>> > <configuration>
>> >
>> > <property>
>> > <name>hadoop.tmp.dir</name>
>> > <value>/tmp/hadoop-${user.name}</value>
>> > <description>A base for other temporary directories</description>
>> > </property>
>> >
>> >
>> > <property>
>> > <name>fs.default.name</name>
>> > <value>hdfs://localhost:54310</value>
>> > <description>The name of the default file system. A URI whose
>> > scheme and authority determine the FileSystem implementation. The
>> > URI's scheme determines the config property (fs.scheme.impl) naming
>> > the FileSystem implementation class. The URI's authority is used to
>> > determine the host, port, etc for a filesystem.</description>
>> > </property>
>> >
>> >
>> > <property>
>> > <name>mapred.job.tracker</name>
>> > <value>localhost:54311</value>
>> > <description>The host and port that the MapReduce job tracker runs at.
>> > If "local", then jobs are run in-process as a single map and
>> > reduce task.</description>
>> > </property>
>> >
>> >
>> > <property>
>> > <name>dfs.replication</name>
>> > <value>1</value>
>> > <description>Default block replication.
>> > The actual number of replications can be specified when the file is
>> > created.
>> > The default is used if replication is not specified in create
>> > time.</description>
>> > </property>
>> > "conf/hadoop-site.xml" 42L, 1271C
>> >
>> >
>> > My hadoop-env.sh looks as follows:
>> >
>> > # Set Hadoop-specific environment variables here.
>> >
>> > # The only required environment variable is JAVA_HOME. All others are
>> > # optional. When running a distributed configuration it is best to
>> > # set JAVA_HOME in this file, so that it is correctly defined on
>> > # remote nodes.
>> >
>> > # The java implementation to use. Required.
>> > export JAVA_HOME=/usr/java/jdk1.6.0_10
>> >
>> > # Extra Java CLASSPATH elements. Optional.
>> > # export HADOOP_CLASSPATH=
>> >
>> > # The maximum amount of heap to use, in MB. Default is 1000.
>> > # export HADOOP_HEAPSIZE=2000
>> >
>> > # Extra Java runtime options. Empty by default.
>> > # export HADOOP_OPTS=-server
>> >
>> > # Command specific options appended to HADOOP_OPTS when specified
>> > export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
>> > $HADOOP_NAMENODE_OPTS"
>> > export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
>> > $HADOOP_SECONDARYNAMENODE_OPTS"
>> > export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
>> > $HADOOP_DATANODE_OPTS"
>> > export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
>> > $HADOOP_BALANCER_OPTS"
>> > export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
>> > $HADOOP_JOBTRACKER_OPTS"
>> > # export HADOOP_TASKTRACKER_OPTS=
>> > # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
>> > # export HADOOP_CLIENT_OPTS
>> >
>> > # Extra ssh options. Empty by default.
>> > # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o
>> SendEnv=HADOOP_CONF_DIR"
>> >
>> > # Where log files are stored. $HADOOP_HOME/logs by default.
>> > # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
>> >
>> > # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
>> > # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
>> >
>> > # host:path where hadoop code should be rsync'd from. Unset by default.
>> > # export HADOOP_MASTER=master:/home/$USER/src/hadoop
>> >
>> > "conf/hadoop-env.sh" 54L, 2236C
>> >
>> > Dont know what the exceptions mean.. Does anyone have an idea?
>> >
>> > THanks
>> > Mithila
>> >
>> >
>> > On Thu, Nov 20, 2008 at 6:42 AM, some speed <sp...@gmail.com>
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > I am working on the same for my master's project and i know how
>> > frustrating
>> > > it can be to get hadoop installed.
>> > > If time is not a factor, I suggest you first try implementing it in a
>> > > psuedo distributed environment. Once you understand how things work by
>> > > implementing a simple map reduce program, u can easily move on to a
>> > cluster.
>> > >
>> > > From what little i know, let me tell u a few things,
>> > >
>> > > I tried using the university network to install hadoop.. it was a real
>> > > pain. mayb it was coz I didnt have the admin privileges( to install
>> HDFS
>> > n
>> > > its files). so make sure u have admin rights or u keep getting an
>> error
>> > of
>> > > port 22 (for ssh) being not opened or the demeons were not started.
>> > > n btw is it conf/start-all.sh?? i think its bin/start -all.sh or
>> > something
>> > > of that sort.
>> > >
>> > > hadoop-site.xml -- i had the links bookmarked somewhere- cant find it
>> > now
>> > > but i think u are supposed to have a few more details in there for a
>> > cluster
>> > > installation. Am sure we can find those online quite easily.
>> > >
>> > > Also i suppose u are using java? if u are good at eclipse, then u can
>> > > implement map reduce/hadoop thru that on a single node (just to get a
>> > hang
>> > > of it).
>> > >
>> > > All the best!
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Wed, Nov 19, 2008 at 6:38 PM, Tom Wheeler <to...@gmail.com>
>> wrote:
>> > >
>> > >> On Wed, Nov 19, 2008 at 5:31 PM, Mithila Nagendra <mn...@asu.edu>
>> > >> wrote:
>> > >> > Oh is that so. Im not sure which UNIX it is since Im working with a
>> > >> cluster
>> > >> > that is remotely accessed.
>> > >>
>> > >> If you can get a shell on the machine, try typing "uname -a" to see
>> > >> what type of UNIX it is.
>> > >>
>> > >> Alternatively, the os.name, os.version and os.arch Java system
>> > >> properties could also help you to identify the operating system.
>> > >>
>> > >> --
>> > >> Tom Wheeler
>> > >> http://www.tomwheeler.com/
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
I tried the 0.18.2 as welll.. it gave me the same exception.. so tried the
lower version.. I should check if this works.. Thanks!
On Fri, Nov 21, 2008 at 5:06 AM, Alex Loddengaard <al...@cloudera.com> wrote:
> Maybe try downloading the Apache Commons - Logging jars (<
> http://commons.apache.org/downloads/download_logging.cgi>) and drop them
> in
> to $HADOOP_HOME/lib.
> Just curious, if you're starting a new cluster, why have you chosen to use
> 0.17.* and not 0.18.2? It would be a good idea to use 0.18.2 if possible.
>
> Alex
>
> On Thu, Nov 20, 2008 at 4:36 PM, Mithila Nagendra <mn...@asu.edu>
> wrote:
>
> > Hey
> > The version is: Linux enpc3740.eas.asu.edu 2.6.9-67.0.20.EL #1 Wed Jun
> 18
> > 12:23:46 EDT 2008 i686 i686 i386 GNU/Linux, this is what I got when I
> used
> > the command uname -a (thanks Tom!)
> >
> > Yea it is bin/start-all.. Following is the exception that I got when i
> > tried
> > to start the daemons..
> >
> >
> > [mithila@node01 mithila]$ ls
> > hadoop-0.17.2.1 hadoop-0.18.2 hadoop-0.18.2.tar.gz
> > [mithila@node01 mithila]$ cd hadoop-0.17*
> > [mithila@node01 hadoop-0.17.2.1]$ ls
> > bin c++ conf docs
> > hadoop-0.17.2.1-examples.jar lib LICENSE.txt NOTICE.txt src
> > build.xml CHANGES.txt contrib hadoop-0.17.2.1-core.jar
> > hadoop-0.17.2.1-test.jar libhdfs logs README.txt webapps
> > [mithila@node01 hadoop-0.17.2.1]$ bin/start-all
> > bash: bin/start-all: No such file or directory
> > [mithila@node01 hadoop-0.17.2.1]$ bin/start-all.sh
> > starting namenode, logging to
> >
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-namenode-node01.out
> > mithila@localhost's password:
> > localhost: starting datanode, logging to
> >
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-datanode-node01.out
> > mithila@localhost's password:
> > localhost: starting secondarynamenode, logging to
> >
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-secondarynamenode-node01.out
> > starting jobtracker, logging to
> >
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-jobtracker-node01.out
> > mithila@localhost's password:
> > localhost: starting tasktracker, logging to
> >
> >
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-tasktracker-node01.out
> > localhost: Exception in thread "main"
> java.lang.ExceptionInInitializerError
> > localhost: Caused by:
> org.apache.commons.logging.LogConfigurationException:
> > User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
> > cannot be found or is not useable.
> > localhost: at
> >
> >
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
> > localhost: at
> >
> >
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
> > localhost: at
> >
> >
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
> > localhost: at
> > org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
> > localhost: at
> > org.apache.hadoop.mapred.TaskTracker.<clinit>(TaskTracker.java:95)
> > localhost: Could not find the main class:
> > org.apache.hadoop.mapred.TaskTracker. Program will exit.
> >
> > AND when I tried formatting the file system I got the following
> exception..
> > I followed Michael Noll s step to install Hadoop.. I m currently working
> on
> > a single node and if this works will move on to multiple nodes in a
> > cluster.
> >
> > [mithila@node01 hadoop-0.17.2.1]$ bin/hadoop namenode -format
> > Exception in thread "main" java.lang.ExceptionInInitializerError
> > Caused by: org.apache.commons.logging.LogConfigurationException:
> > User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
> > cannot be found or is not useable.
> > at
> >
> >
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
> > at
> >
> >
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
> > at
> >
> >
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
> > at
> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
> > at org.apache.hadoop.dfs.NameNode.<clinit>(NameNode.java:88)
> > Could not find the main class: org.apache.hadoop.dfs.NameNode. Program
> > will
> > exit.
> >
> >
> > I have no idea whats wrong... my hadoop-xml file looks as follows:
> >
> > <?xml version="1.0"?>
> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >
> > <!-- Put site-specific property overrides in this file. -->
> >
> > <configuration>
> >
> > <property>
> > <name>hadoop.tmp.dir</name>
> > <value>/tmp/hadoop-${user.name}</value>
> > <description>A base for other temporary directories</description>
> > </property>
> >
> >
> > <property>
> > <name>fs.default.name</name>
> > <value>hdfs://localhost:54310</value>
> > <description>The name of the default file system. A URI whose
> > scheme and authority determine the FileSystem implementation. The
> > URI's scheme determines the config property (fs.scheme.impl) naming
> > the FileSystem implementation class. The URI's authority is used to
> > determine the host, port, etc for a filesystem.</description>
> > </property>
> >
> >
> > <property>
> > <name>mapred.job.tracker</name>
> > <value>localhost:54311</value>
> > <description>The host and port that the MapReduce job tracker runs at.
> > If "local", then jobs are run in-process as a single map and
> > reduce task.</description>
> > </property>
> >
> >
> > <property>
> > <name>dfs.replication</name>
> > <value>1</value>
> > <description>Default block replication.
> > The actual number of replications can be specified when the file is
> > created.
> > The default is used if replication is not specified in create
> > time.</description>
> > </property>
> > "conf/hadoop-site.xml" 42L, 1271C
> >
> >
> > My hadoop-env.sh looks as follows:
> >
> > # Set Hadoop-specific environment variables here.
> >
> > # The only required environment variable is JAVA_HOME. All others are
> > # optional. When running a distributed configuration it is best to
> > # set JAVA_HOME in this file, so that it is correctly defined on
> > # remote nodes.
> >
> > # The java implementation to use. Required.
> > export JAVA_HOME=/usr/java/jdk1.6.0_10
> >
> > # Extra Java CLASSPATH elements. Optional.
> > # export HADOOP_CLASSPATH=
> >
> > # The maximum amount of heap to use, in MB. Default is 1000.
> > # export HADOOP_HEAPSIZE=2000
> >
> > # Extra Java runtime options. Empty by default.
> > # export HADOOP_OPTS=-server
> >
> > # Command specific options appended to HADOOP_OPTS when specified
> > export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> > $HADOOP_NAMENODE_OPTS"
> > export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
> > $HADOOP_SECONDARYNAMENODE_OPTS"
> > export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
> > $HADOOP_DATANODE_OPTS"
> > export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
> > $HADOOP_BALANCER_OPTS"
> > export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
> > $HADOOP_JOBTRACKER_OPTS"
> > # export HADOOP_TASKTRACKER_OPTS=
> > # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
> > # export HADOOP_CLIENT_OPTS
> >
> > # Extra ssh options. Empty by default.
> > # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
> >
> > # Where log files are stored. $HADOOP_HOME/logs by default.
> > # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
> >
> > # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
> > # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
> >
> > # host:path where hadoop code should be rsync'd from. Unset by default.
> > # export HADOOP_MASTER=master:/home/$USER/src/hadoop
> >
> > "conf/hadoop-env.sh" 54L, 2236C
> >
> > Dont know what the exceptions mean.. Does anyone have an idea?
> >
> > THanks
> > Mithila
> >
> >
> > On Thu, Nov 20, 2008 at 6:42 AM, some speed <sp...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > I am working on the same for my master's project and i know how
> > frustrating
> > > it can be to get hadoop installed.
> > > If time is not a factor, I suggest you first try implementing it in a
> > > psuedo distributed environment. Once you understand how things work by
> > > implementing a simple map reduce program, u can easily move on to a
> > cluster.
> > >
> > > From what little i know, let me tell u a few things,
> > >
> > > I tried using the university network to install hadoop.. it was a real
> > > pain. mayb it was coz I didnt have the admin privileges( to install
> HDFS
> > n
> > > its files). so make sure u have admin rights or u keep getting an error
> > of
> > > port 22 (for ssh) being not opened or the demeons were not started.
> > > n btw is it conf/start-all.sh?? i think its bin/start -all.sh or
> > something
> > > of that sort.
> > >
> > > hadoop-site.xml -- i had the links bookmarked somewhere- cant find it
> > now
> > > but i think u are supposed to have a few more details in there for a
> > cluster
> > > installation. Am sure we can find those online quite easily.
> > >
> > > Also i suppose u are using java? if u are good at eclipse, then u can
> > > implement map reduce/hadoop thru that on a single node (just to get a
> > hang
> > > of it).
> > >
> > > All the best!
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Nov 19, 2008 at 6:38 PM, Tom Wheeler <to...@gmail.com>
> wrote:
> > >
> > >> On Wed, Nov 19, 2008 at 5:31 PM, Mithila Nagendra <mn...@asu.edu>
> > >> wrote:
> > >> > Oh is that so. Im not sure which UNIX it is since Im working with a
> > >> cluster
> > >> > that is remotely accessed.
> > >>
> > >> If you can get a shell on the machine, try typing "uname -a" to see
> > >> what type of UNIX it is.
> > >>
> > >> Alternatively, the os.name, os.version and os.arch Java system
> > >> properties could also help you to identify the operating system.
> > >>
> > >> --
> > >> Tom Wheeler
> > >> http://www.tomwheeler.com/
> > >>
> > >
> > >
> >
>
Re: Hadoop Installation
Posted by Alex Loddengaard <al...@cloudera.com>.
Maybe try downloading the Apache Commons - Logging jars (<
http://commons.apache.org/downloads/download_logging.cgi>) and drop them in
to $HADOOP_HOME/lib.
Just curious, if you're starting a new cluster, why have you chosen to use
0.17.* and not 0.18.2? It would be a good idea to use 0.18.2 if possible.
Alex
On Thu, Nov 20, 2008 at 4:36 PM, Mithila Nagendra <mn...@asu.edu> wrote:
> Hey
> The version is: Linux enpc3740.eas.asu.edu 2.6.9-67.0.20.EL #1 Wed Jun 18
> 12:23:46 EDT 2008 i686 i686 i386 GNU/Linux, this is what I got when I used
> the command uname -a (thanks Tom!)
>
> Yea it is bin/start-all.. Following is the exception that I got when i
> tried
> to start the daemons..
>
>
> [mithila@node01 mithila]$ ls
> hadoop-0.17.2.1 hadoop-0.18.2 hadoop-0.18.2.tar.gz
> [mithila@node01 mithila]$ cd hadoop-0.17*
> [mithila@node01 hadoop-0.17.2.1]$ ls
> bin c++ conf docs
> hadoop-0.17.2.1-examples.jar lib LICENSE.txt NOTICE.txt src
> build.xml CHANGES.txt contrib hadoop-0.17.2.1-core.jar
> hadoop-0.17.2.1-test.jar libhdfs logs README.txt webapps
> [mithila@node01 hadoop-0.17.2.1]$ bin/start-all
> bash: bin/start-all: No such file or directory
> [mithila@node01 hadoop-0.17.2.1]$ bin/start-all.sh
> starting namenode, logging to
>
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-namenode-node01.out
> mithila@localhost's password:
> localhost: starting datanode, logging to
>
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-datanode-node01.out
> mithila@localhost's password:
> localhost: starting secondarynamenode, logging to
>
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-secondarynamenode-node01.out
> starting jobtracker, logging to
>
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-jobtracker-node01.out
> mithila@localhost's password:
> localhost: starting tasktracker, logging to
>
> /home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-tasktracker-node01.out
> localhost: Exception in thread "main" java.lang.ExceptionInInitializerError
> localhost: Caused by: org.apache.commons.logging.LogConfigurationException:
> User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
> cannot be found or is not useable.
> localhost: at
>
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
> localhost: at
>
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
> localhost: at
>
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
> localhost: at
> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
> localhost: at
> org.apache.hadoop.mapred.TaskTracker.<clinit>(TaskTracker.java:95)
> localhost: Could not find the main class:
> org.apache.hadoop.mapred.TaskTracker. Program will exit.
>
> AND when I tried formatting the file system I got the following exception..
> I followed Michael Noll s step to install Hadoop.. I m currently working on
> a single node and if this works will move on to multiple nodes in a
> cluster.
>
> [mithila@node01 hadoop-0.17.2.1]$ bin/hadoop namenode -format
> Exception in thread "main" java.lang.ExceptionInInitializerError
> Caused by: org.apache.commons.logging.LogConfigurationException:
> User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
> cannot be found or is not useable.
> at
>
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
> at
>
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
> at
>
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
> at org.apache.hadoop.dfs.NameNode.<clinit>(NameNode.java:88)
> Could not find the main class: org.apache.hadoop.dfs.NameNode. Program
> will
> exit.
>
>
> I have no idea whats wrong... my hadoop-xml file looks as follows:
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/tmp/hadoop-${user.name}</value>
> <description>A base for other temporary directories</description>
> </property>
>
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://localhost:54310</value>
> <description>The name of the default file system. A URI whose
> scheme and authority determine the FileSystem implementation. The
> URI's scheme determines the config property (fs.scheme.impl) naming
> the FileSystem implementation class. The URI's authority is used to
> determine the host, port, etc for a filesystem.</description>
> </property>
>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>localhost:54311</value>
> <description>The host and port that the MapReduce job tracker runs at.
> If "local", then jobs are run in-process as a single map and
> reduce task.</description>
> </property>
>
>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> <description>Default block replication.
> The actual number of replications can be specified when the file is
> created.
> The default is used if replication is not specified in create
> time.</description>
> </property>
> "conf/hadoop-site.xml" 42L, 1271C
>
>
> My hadoop-env.sh looks as follows:
>
> # Set Hadoop-specific environment variables here.
>
> # The only required environment variable is JAVA_HOME. All others are
> # optional. When running a distributed configuration it is best to
> # set JAVA_HOME in this file, so that it is correctly defined on
> # remote nodes.
>
> # The java implementation to use. Required.
> export JAVA_HOME=/usr/java/jdk1.6.0_10
>
> # Extra Java CLASSPATH elements. Optional.
> # export HADOOP_CLASSPATH=
>
> # The maximum amount of heap to use, in MB. Default is 1000.
> # export HADOOP_HEAPSIZE=2000
>
> # Extra Java runtime options. Empty by default.
> # export HADOOP_OPTS=-server
>
> # Command specific options appended to HADOOP_OPTS when specified
> export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_NAMENODE_OPTS"
> export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_SECONDARYNAMENODE_OPTS"
> export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_DATANODE_OPTS"
> export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_BALANCER_OPTS"
> export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_JOBTRACKER_OPTS"
> # export HADOOP_TASKTRACKER_OPTS=
> # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
> # export HADOOP_CLIENT_OPTS
>
> # Extra ssh options. Empty by default.
> # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
>
> # Where log files are stored. $HADOOP_HOME/logs by default.
> # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
>
> # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
> # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
>
> # host:path where hadoop code should be rsync'd from. Unset by default.
> # export HADOOP_MASTER=master:/home/$USER/src/hadoop
>
> "conf/hadoop-env.sh" 54L, 2236C
>
> Dont know what the exceptions mean.. Does anyone have an idea?
>
> THanks
> Mithila
>
>
> On Thu, Nov 20, 2008 at 6:42 AM, some speed <sp...@gmail.com> wrote:
>
> > Hi,
> >
> > I am working on the same for my master's project and i know how
> frustrating
> > it can be to get hadoop installed.
> > If time is not a factor, I suggest you first try implementing it in a
> > psuedo distributed environment. Once you understand how things work by
> > implementing a simple map reduce program, u can easily move on to a
> cluster.
> >
> > From what little i know, let me tell u a few things,
> >
> > I tried using the university network to install hadoop.. it was a real
> > pain. mayb it was coz I didnt have the admin privileges( to install HDFS
> n
> > its files). so make sure u have admin rights or u keep getting an error
> of
> > port 22 (for ssh) being not opened or the demeons were not started.
> > n btw is it conf/start-all.sh?? i think its bin/start -all.sh or
> something
> > of that sort.
> >
> > hadoop-site.xml -- i had the links bookmarked somewhere- cant find it
> now
> > but i think u are supposed to have a few more details in there for a
> cluster
> > installation. Am sure we can find those online quite easily.
> >
> > Also i suppose u are using java? if u are good at eclipse, then u can
> > implement map reduce/hadoop thru that on a single node (just to get a
> hang
> > of it).
> >
> > All the best!
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Nov 19, 2008 at 6:38 PM, Tom Wheeler <to...@gmail.com> wrote:
> >
> >> On Wed, Nov 19, 2008 at 5:31 PM, Mithila Nagendra <mn...@asu.edu>
> >> wrote:
> >> > Oh is that so. Im not sure which UNIX it is since Im working with a
> >> cluster
> >> > that is remotely accessed.
> >>
> >> If you can get a shell on the machine, try typing "uname -a" to see
> >> what type of UNIX it is.
> >>
> >> Alternatively, the os.name, os.version and os.arch Java system
> >> properties could also help you to identify the operating system.
> >>
> >> --
> >> Tom Wheeler
> >> http://www.tomwheeler.com/
> >>
> >
> >
>
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
Hey
The version is: Linux enpc3740.eas.asu.edu 2.6.9-67.0.20.EL #1 Wed Jun 18
12:23:46 EDT 2008 i686 i686 i386 GNU/Linux, this is what I got when I used
the command uname -a (thanks Tom!)
Yea it is bin/start-all.. Following is the exception that I got when i tried
to start the daemons..
[mithila@node01 mithila]$ ls
hadoop-0.17.2.1 hadoop-0.18.2 hadoop-0.18.2.tar.gz
[mithila@node01 mithila]$ cd hadoop-0.17*
[mithila@node01 hadoop-0.17.2.1]$ ls
bin c++ conf docs
hadoop-0.17.2.1-examples.jar lib LICENSE.txt NOTICE.txt src
build.xml CHANGES.txt contrib hadoop-0.17.2.1-core.jar
hadoop-0.17.2.1-test.jar libhdfs logs README.txt webapps
[mithila@node01 hadoop-0.17.2.1]$ bin/start-all
bash: bin/start-all: No such file or directory
[mithila@node01 hadoop-0.17.2.1]$ bin/start-all.sh
starting namenode, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-namenode-node01.out
mithila@localhost's password:
localhost: starting datanode, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-datanode-node01.out
mithila@localhost's password:
localhost: starting secondarynamenode, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-secondarynamenode-node01.out
starting jobtracker, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-jobtracker-node01.out
mithila@localhost's password:
localhost: starting tasktracker, logging to
/home/mithila/hadoop-0.17.2.1/bin/../logs/hadoop-mithila-tasktracker-node01.out
localhost: Exception in thread "main" java.lang.ExceptionInInitializerError
localhost: Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
cannot be found or is not useable.
localhost: at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
localhost: at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
localhost: at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
localhost: at
org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
localhost: at
org.apache.hadoop.mapred.TaskTracker.<clinit>(TaskTracker.java:95)
localhost: Could not find the main class:
org.apache.hadoop.mapred.TaskTracker. Program will exit.
AND when I tried formatting the file system I got the following exception..
I followed Michael Noll s step to install Hadoop.. I m currently working on
a single node and if this works will move on to multiple nodes in a cluster.
[mithila@node01 hadoop-0.17.2.1]$ bin/hadoop namenode -format
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: org.apache.commons.logging.LogConfigurationException:
User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
cannot be found or is not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:704)
at org.apache.hadoop.dfs.NameNode.<clinit>(NameNode.java:88)
Could not find the main class: org.apache.hadoop.dfs.NameNode. Program will
exit.
I have no idea whats wrong... my hadoop-xml file looks as follows:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-${user.name}</value>
<description>A base for other temporary directories</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
URI's scheme determines the config property (fs.scheme.impl) naming
the FileSystem implementation class. The URI's authority is used to
determine the host, port, etc for a filesystem.</description>
</property>
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<description>The host and port that the MapReduce job tracker runs at.
If "local", then jobs are run in-process as a single map and
reduce task.</description>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create
time.</description>
</property>
"conf/hadoop-site.xml" 42L, 1271C
My hadoop-env.sh looks as follows:
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use. Required.
export JAVA_HOME=/usr/java/jdk1.6.0_10
# Extra Java CLASSPATH elements. Optional.
# export HADOOP_CLASSPATH=
# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000
# Extra Java runtime options. Empty by default.
# export HADOOP_OPTS=-server
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
$HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS
# Extra ssh options. Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
# File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
# host:path where hadoop code should be rsync'd from. Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop
"conf/hadoop-env.sh" 54L, 2236C
Dont know what the exceptions mean.. Does anyone have an idea?
THanks
Mithila
On Thu, Nov 20, 2008 at 6:42 AM, some speed <sp...@gmail.com> wrote:
> Hi,
>
> I am working on the same for my master's project and i know how frustrating
> it can be to get hadoop installed.
> If time is not a factor, I suggest you first try implementing it in a
> psuedo distributed environment. Once you understand how things work by
> implementing a simple map reduce program, u can easily move on to a cluster.
>
> From what little i know, let me tell u a few things,
>
> I tried using the university network to install hadoop.. it was a real
> pain. mayb it was coz I didnt have the admin privileges( to install HDFS n
> its files). so make sure u have admin rights or u keep getting an error of
> port 22 (for ssh) being not opened or the demeons were not started.
> n btw is it conf/start-all.sh?? i think its bin/start -all.sh or something
> of that sort.
>
> hadoop-site.xml -- i had the links bookmarked somewhere- cant find it now
> but i think u are supposed to have a few more details in there for a cluster
> installation. Am sure we can find those online quite easily.
>
> Also i suppose u are using java? if u are good at eclipse, then u can
> implement map reduce/hadoop thru that on a single node (just to get a hang
> of it).
>
> All the best!
>
>
>
>
>
>
>
>
>
> On Wed, Nov 19, 2008 at 6:38 PM, Tom Wheeler <to...@gmail.com> wrote:
>
>> On Wed, Nov 19, 2008 at 5:31 PM, Mithila Nagendra <mn...@asu.edu>
>> wrote:
>> > Oh is that so. Im not sure which UNIX it is since Im working with a
>> cluster
>> > that is remotely accessed.
>>
>> If you can get a shell on the machine, try typing "uname -a" to see
>> what type of UNIX it is.
>>
>> Alternatively, the os.name, os.version and os.arch Java system
>> properties could also help you to identify the operating system.
>>
>> --
>> Tom Wheeler
>> http://www.tomwheeler.com/
>>
>
>
Re: Hadoop Installation
Posted by Tom Wheeler <to...@gmail.com>.
On Wed, Nov 19, 2008 at 5:31 PM, Mithila Nagendra <mn...@asu.edu> wrote:
> Oh is that so. Im not sure which UNIX it is since Im working with a cluster
> that is remotely accessed.
If you can get a shell on the machine, try typing "uname -a" to see
what type of UNIX it is.
Alternatively, the os.name, os.version and os.arch Java system
properties could also help you to identify the operating system.
--
Tom Wheeler
http://www.tomwheeler.com/
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
Oh is that so. Im not sure which UNIX it is since Im working with a cluster
that is remotely accessed. I will send the error out, but its gotta do with
Log4Jogger class...
On Wed, Nov 19, 2008 at 10:48 PM, Alex Loddengaard <al...@cloudera.com>wrote:
> None of your emails have had attachments. I think this list might strip
> them. Can you copy-paste the error? Though I think the error won't be
> useful. I'm pretty confident your issue is with Java. What UNIX are you
> using?
> Alex
>
> On Wed, Nov 19, 2008 at 11:38 AM, Mithila Nagendra <mn...@asu.edu>
> wrote:
>
> > Hey
> > My hadoop version is 0.17.0.. check out the screen shots i ve put in..
> >
> > Mithila
> >
> >
> > On Wed, Nov 19, 2008 at 9:29 PM, Sagar Naik <sn...@attributor.com>
> wrote:
> >
> >> Mithila Nagendra wrote:
> >>
> >>> Hello
> >>> I m currently a student at Arizona State University, Tempe, Arizona,
> >>> pursuing my masters in Computer Science. I m currently involved in a
> >>> research project that makes use of Hadoop to run various map reduce
> >>> functions. Hence I searched the web on whats the best way to install
> >>> Hadoop
> >>> on different nodes in a cluster, and I stumbled upon your website.
> >>>
> >>> I used your tutorial on "How to install hadoop on a linux system" by
> >>> Michael
> >>> Noll to setup Hadoop on a UNIX system. I have a few questions related
> to
> >>> it:
> >>>
> >>> 1. Does it matter that I m installing hadoop on UNIX and not on LINUX -
> >>> do I
> >>> have to follow different steps?
> >>> 2. The configuration for hadoop-site.xml - does it remain the same no
> >>> matter
> >>> what platform is being used? Do I just type the same thing out in the
> >>> file
> >>> hadoop-site.xml present in the hadoop installation on the node?
> >>> 3. When I try to start the daemons by executing the command
> >>> conf/start-all.sh, I get an exception which says "hadoop: user
> specified
> >>> log
> >>> class 'org.apache.commons.logging.impl.Log4JLogger' cannot be found or
> is
> >>> not usable" - this happens when tasktracker is being started. What
> steps
> >>> do
> >>> I take to deal with this exception?
> >>>
> >>>
> >>>
> >> start-all is in {HADOOP_HOME}/bin/ . What is your hadoop version ?
> >>
> >>
> >>
> >> I could send you the screen shot of the exception if you wish. It would
> >>> be
> >>> of immense help if you could provide answers for the above questions.
> >>>
> >>> Thank you! Looking forward to your reply.
> >>>
> >>> Best Regards
> >>> Mithila Nagendra
> >>>
> >>>
> >>>
> >>
> >>
> >
>
Re: Hadoop Installation
Posted by Alex Loddengaard <al...@cloudera.com>.
None of your emails have had attachments. I think this list might strip
them. Can you copy-paste the error? Though I think the error won't be
useful. I'm pretty confident your issue is with Java. What UNIX are you
using?
Alex
On Wed, Nov 19, 2008 at 11:38 AM, Mithila Nagendra <mn...@asu.edu> wrote:
> Hey
> My hadoop version is 0.17.0.. check out the screen shots i ve put in..
>
> Mithila
>
>
> On Wed, Nov 19, 2008 at 9:29 PM, Sagar Naik <sn...@attributor.com> wrote:
>
>> Mithila Nagendra wrote:
>>
>>> Hello
>>> I m currently a student at Arizona State University, Tempe, Arizona,
>>> pursuing my masters in Computer Science. I m currently involved in a
>>> research project that makes use of Hadoop to run various map reduce
>>> functions. Hence I searched the web on whats the best way to install
>>> Hadoop
>>> on different nodes in a cluster, and I stumbled upon your website.
>>>
>>> I used your tutorial on "How to install hadoop on a linux system" by
>>> Michael
>>> Noll to setup Hadoop on a UNIX system. I have a few questions related to
>>> it:
>>>
>>> 1. Does it matter that I m installing hadoop on UNIX and not on LINUX -
>>> do I
>>> have to follow different steps?
>>> 2. The configuration for hadoop-site.xml - does it remain the same no
>>> matter
>>> what platform is being used? Do I just type the same thing out in the
>>> file
>>> hadoop-site.xml present in the hadoop installation on the node?
>>> 3. When I try to start the daemons by executing the command
>>> conf/start-all.sh, I get an exception which says "hadoop: user specified
>>> log
>>> class 'org.apache.commons.logging.impl.Log4JLogger' cannot be found or is
>>> not usable" - this happens when tasktracker is being started. What steps
>>> do
>>> I take to deal with this exception?
>>>
>>>
>>>
>> start-all is in {HADOOP_HOME}/bin/ . What is your hadoop version ?
>>
>>
>>
>> I could send you the screen shot of the exception if you wish. It would
>>> be
>>> of immense help if you could provide answers for the above questions.
>>>
>>> Thank you! Looking forward to your reply.
>>>
>>> Best Regards
>>> Mithila Nagendra
>>>
>>>
>>>
>>
>>
>
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
Hey
My hadoop version is 0.17.0.. check out the screen shots i ve put in..
Mithila
On Wed, Nov 19, 2008 at 9:29 PM, Sagar Naik <sn...@attributor.com> wrote:
> Mithila Nagendra wrote:
>
>> Hello
>> I m currently a student at Arizona State University, Tempe, Arizona,
>> pursuing my masters in Computer Science. I m currently involved in a
>> research project that makes use of Hadoop to run various map reduce
>> functions. Hence I searched the web on whats the best way to install
>> Hadoop
>> on different nodes in a cluster, and I stumbled upon your website.
>>
>> I used your tutorial on "How to install hadoop on a linux system" by
>> Michael
>> Noll to setup Hadoop on a UNIX system. I have a few questions related to
>> it:
>>
>> 1. Does it matter that I m installing hadoop on UNIX and not on LINUX - do
>> I
>> have to follow different steps?
>> 2. The configuration for hadoop-site.xml - does it remain the same no
>> matter
>> what platform is being used? Do I just type the same thing out in the file
>> hadoop-site.xml present in the hadoop installation on the node?
>> 3. When I try to start the daemons by executing the command
>> conf/start-all.sh, I get an exception which says "hadoop: user specified
>> log
>> class 'org.apache.commons.logging.impl.Log4JLogger' cannot be found or is
>> not usable" - this happens when tasktracker is being started. What steps
>> do
>> I take to deal with this exception?
>>
>>
>>
> start-all is in {HADOOP_HOME}/bin/ . What is your hadoop version ?
>
>
>
> I could send you the screen shot of the exception if you wish. It would be
>> of immense help if you could provide answers for the above questions.
>>
>> Thank you! Looking forward to your reply.
>>
>> Best Regards
>> Mithila Nagendra
>>
>>
>>
>
>
Re: Hadoop Installation
Posted by Sagar Naik <sn...@attributor.com>.
Mithila Nagendra wrote:
> Hello
> I m currently a student at Arizona State University, Tempe, Arizona,
> pursuing my masters in Computer Science. I m currently involved in a
> research project that makes use of Hadoop to run various map reduce
> functions. Hence I searched the web on whats the best way to install Hadoop
> on different nodes in a cluster, and I stumbled upon your website.
>
> I used your tutorial on "How to install hadoop on a linux system" by Michael
> Noll to setup Hadoop on a UNIX system. I have a few questions related to it:
>
> 1. Does it matter that I m installing hadoop on UNIX and not on LINUX - do I
> have to follow different steps?
> 2. The configuration for hadoop-site.xml - does it remain the same no matter
> what platform is being used? Do I just type the same thing out in the file
> hadoop-site.xml present in the hadoop installation on the node?
> 3. When I try to start the daemons by executing the command
> conf/start-all.sh, I get an exception which says "hadoop: user specified log
> class 'org.apache.commons.logging.impl.Log4JLogger' cannot be found or is
> not usable" - this happens when tasktracker is being started. What steps do
> I take to deal with this exception?
>
>
start-all is in {HADOOP_HOME}/bin/ . What is your hadoop version ?
> I could send you the screen shot of the exception if you wish. It would be
> of immense help if you could provide answers for the above questions.
>
> Thank you! Looking forward to your reply.
>
> Best Regards
> Mithila Nagendra
>
>
Re: Hadoop Installation
Posted by Alex Loddengaard <al...@cloudera.com>.
By "UNIX" do you mean FreeBSD? The Hadoop configuration is platform
agnostic, so your issue is probably related to your Java configuration
(classpath, etc).
Alex
On Wed, Nov 19, 2008 at 10:20 AM, Mithila Nagendra <mn...@asu.edu> wrote:
> I ve attached the screen shots of the exception and hadoop-site.xml.
> Thanks!
>
>
> On Wed, Nov 19, 2008 at 9:12 PM, Mithila Nagendra <mn...@asu.edu>wrote:
>
>> Hello
>> I m currently a student at Arizona State University, Tempe, Arizona,
>> pursuing my masters in Computer Science. I m currently involved in a
>> research project that makes use of Hadoop to run various map reduce
>> functions. Hence I searched the web on whats the best way to install Hadoop
>> on different nodes in a cluster, and I stumbled upon your website.
>>
>> I used your tutorial on "How to install hadoop on a linux system" by
>> Michael Noll to setup Hadoop on a UNIX system. I have a few questions
>> related to it:
>>
>> 1. Does it matter that I m installing hadoop on UNIX and not on LINUX - do
>> I have to follow different steps?
>> 2. The configuration for hadoop-site.xml - does it remain the same no
>> matter what platform is being used? Do I just type the same thing out in the
>> file hadoop-site.xml present in the hadoop installation on the node?
>> 3. When I try to start the daemons by executing the command
>> conf/start-all.sh, I get an exception which says "hadoop: user specified
>> log class 'org.apache.commons.logging.impl.Log4JLogger' cannot be found or
>> is not usable" - this happens when tasktracker is being started. What steps
>> do I take to deal with this exception?
>>
>> I could send you the screen shot of the exception if you wish. It would be
>> of immense help if you could provide answers for the above questions.
>>
>> Thank you! Looking forward to your reply.
>>
>> Best Regards
>> Mithila Nagendra
>>
>
>
Re: Hadoop Installation
Posted by Mithila Nagendra <mn...@asu.edu>.
I ve attached the screen shots of the exception and hadoop-site.xml.
Thanks!
On Wed, Nov 19, 2008 at 9:12 PM, Mithila Nagendra <mn...@asu.edu> wrote:
> Hello
> I m currently a student at Arizona State University, Tempe, Arizona,
> pursuing my masters in Computer Science. I m currently involved in a
> research project that makes use of Hadoop to run various map reduce
> functions. Hence I searched the web on whats the best way to install Hadoop
> on different nodes in a cluster, and I stumbled upon your website.
>
> I used your tutorial on "How to install hadoop on a linux system" by
> Michael Noll to setup Hadoop on a UNIX system. I have a few questions
> related to it:
>
> 1. Does it matter that I m installing hadoop on UNIX and not on LINUX - do
> I have to follow different steps?
> 2. The configuration for hadoop-site.xml - does it remain the same no
> matter what platform is being used? Do I just type the same thing out in the
> file hadoop-site.xml present in the hadoop installation on the node?
> 3. When I try to start the daemons by executing the command
> conf/start-all.sh, I get an exception which says "hadoop: user specified
> log class 'org.apache.commons.logging.impl.Log4JLogger' cannot be found or
> is not usable" - this happens when tasktracker is being started. What steps
> do I take to deal with this exception?
>
> I could send you the screen shot of the exception if you wish. It would be
> of immense help if you could provide answers for the above questions.
>
> Thank you! Looking forward to your reply.
>
> Best Regards
> Mithila Nagendra
>