You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Amy Zhang <am...@gmail.com> on 2011/01/24 06:24:47 UTC
Why does my HDFS point to my current directory?
A few months ago, we installed Cloudera Hadoop 3 in our local machine
and everything was fine. Recently we also installed Whirr to start
working with clusters. Although we faced some problems, after a while,
we can start up a cluster, log into its master node and commence work.
However, I found out recently that when I type:
hadoop dfs -ls
into our local machine, it now displays everything in the current
directory I am in, not the contents of the DFS. This didn't use to
happen, so we are thinking something got messed up when we installed
Whirr.
What could have caused this, and more importantly, how can we get our
local hadoop dfs to point to the correct location?
I posted this question on a Cloudera support board and I got the
follow response:
For your question, you should email the Whirr user list at
whirr-user@incubator.apache.org. I suspect you have some old jar files
on your classpath, but they'll know for sure.
I appreciate the help and time.
Thank you,
Amy Zhang
Re: Why does my HDFS point to my current directory?
Posted by Amy Zhang <am...@gmail.com>.
Thank you Philip!
On Mon, Jan 24, 2011 at 12:47 AM, Philip Zeyliger <ph...@cloudera.com> wrote:
>
>
> On Sun, Jan 23, 2011 at 9:42 PM, Amy Zhang <am...@gmail.com> wrote:
>>
>> Ok, it definitely looks like something got mixed up. Inside
>> /etc/hadoop/ I have conf, conf.empty, conf.pseudo, and conf.whirr.
>> The contents of conf are
>>
>> [amyzhang@cactuar conf]$ ls
>> capacity-scheduler.xml hadoop-policy.xml ssl-client.xml.example
>> configuration.xsl log4j.properties ssl-server.xml.example
>> fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg
>> hadoop-env.sh masters whirr.log
>> hadoop-metrics.properties slaves whirr.log.2010-12-23
>>
>> This is exactly the files I have in conf.whirr. So it seems my files
>> from conf.whirr replaced the original files in conf.
>> These are the files I have in conf.pseudo and conf.empty:
>>
>> [amyzhang@cactuar conf.pseudo]$ ls
>> capacity-scheduler.xml hadoop-policy.xml slaves
>> configuration.xsl hdfs-site.xml ssl-client.xml.example
>> core-site.xml log4j.properties ssl-server.xml.example
>> fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg
>> hadoop-env.sh mapred-site.xml
>> hadoop-metrics.properties masters
>>
>> What should I do at this point?
>
> /etc/hadoop/conf is actually a symlink to /etc/hadoop/conf.whirr. (Most
> likely.) This is managed by a system called 'alternatives', which is
> described
> at https://wiki.cloudera.com/display/DOC/CDH3+Deployment+on+a+Cluster.
> Are you trying to access your pseudo-distributed (local) cluster? If so,
> either update alternatives to use conf.pseudo (see the above page for how to
> do that), or, tell hadoop to use a certain directory explicitly by setting
> HADOOP_CONF_DIR=/etc/hadoop/conf.pseudo as an environment variable.
> Cheers,
> -- Philip
>
>>
>> -Amy
>>
>>
>>
>> On Mon, Jan 24, 2011 at 12:28 AM, Philip Zeyliger <ph...@cloudera.com>
>> wrote:
>> > "hadoop dfs -ls" reads the configuration "fs.default.name" from
>> > somewhere,
>> > namely a copy of core-site.xml or hdfs-site.xml on your classpath.
>> > Typically, "hadoop dfs" looks in /etc/hadoop/conf, especially if you're
>> > using CDH. The default setting for fs.default.name is "file:///", which
>> > translates to your local file system. Perhaps you've lost that config
>> > file
>> > somewhere along the way?
>> > -- Philip
>> >
>> > On Sun, Jan 23, 2011 at 9:24 PM, Amy Zhang <am...@gmail.com>
>> > wrote:
>> >>
>> >> A few months ago, we installed Cloudera Hadoop 3 in our local machine
>> >> and everything was fine. Recently we also installed Whirr to start
>> >> working with clusters. Although we faced some problems, after a while,
>> >> we can start up a cluster, log into its master node and commence work.
>> >> However, I found out recently that when I type:
>> >>
>> >> hadoop dfs -ls
>> >>
>> >> into our local machine, it now displays everything in the current
>> >> directory I am in, not the contents of the DFS. This didn't use to
>> >> happen, so we are thinking something got messed up when we installed
>> >> Whirr.
>> >>
>> >> What could have caused this, and more importantly, how can we get our
>> >> local hadoop dfs to point to the correct location?
>> >>
>> >> I posted this question on a Cloudera support board and I got the
>> >> follow response:
>> >>
>> >> For your question, you should email the Whirr user list at
>> >> whirr-user@incubator.apache.org. I suspect you have some old jar files
>> >> on your classpath, but they'll know for sure.
>> >>
>> >> I appreciate the help and time.
>> >>
>> >> Thank you,
>> >> Amy Zhang
>> >
>> >
>
>
Re: Why does my HDFS point to my current directory?
Posted by Philip Zeyliger <ph...@cloudera.com>.
On Sun, Jan 23, 2011 at 9:42 PM, Amy Zhang <am...@gmail.com> wrote:
> Ok, it definitely looks like something got mixed up. Inside
> /etc/hadoop/ I have conf, conf.empty, conf.pseudo, and conf.whirr.
> The contents of conf are
>
> [amyzhang@cactuar conf]$ ls
> capacity-scheduler.xml hadoop-policy.xml ssl-client.xml.example
> configuration.xsl log4j.properties ssl-server.xml.example
> fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg
> hadoop-env.sh masters whirr.log
> hadoop-metrics.properties slaves whirr.log.2010-12-23
>
> This is exactly the files I have in conf.whirr. So it seems my files
> from conf.whirr replaced the original files in conf.
> These are the files I have in conf.pseudo and conf.empty:
>
> [amyzhang@cactuar conf.pseudo]$ ls
> capacity-scheduler.xml hadoop-policy.xml slaves
> configuration.xsl hdfs-site.xml ssl-client.xml.example
> core-site.xml log4j.properties ssl-server.xml.example
> fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg
> hadoop-env.sh mapred-site.xml
> hadoop-metrics.properties masters
>
> What should I do at this point?
>
/etc/hadoop/conf is actually a symlink to /etc/hadoop/conf.whirr. (Most
likely.) This is managed by a system called 'alternatives', which is
described at
https://wiki.cloudera.com/display/DOC/CDH3+Deployment+on+a+Cluster.
Are you trying to access your pseudo-distributed (local) cluster? If so,
either update alternatives to use conf.pseudo (see the above page for how to
do that), or, tell hadoop to use a certain directory explicitly by setting
HADOOP_CONF_DIR=/etc/hadoop/conf.pseudo as an environment variable.
Cheers,
-- Philip
>
> -Amy
>
>
>
> On Mon, Jan 24, 2011 at 12:28 AM, Philip Zeyliger <ph...@cloudera.com>
> wrote:
> > "hadoop dfs -ls" reads the configuration "fs.default.name" from
> somewhere,
> > namely a copy of core-site.xml or hdfs-site.xml on your classpath.
> > Typically, "hadoop dfs" looks in /etc/hadoop/conf, especially if you're
> > using CDH. The default setting for fs.default.name is "file:///", which
> > translates to your local file system. Perhaps you've lost that config
> file
> > somewhere along the way?
> > -- Philip
> >
> > On Sun, Jan 23, 2011 at 9:24 PM, Amy Zhang <am...@gmail.com>
> wrote:
> >>
> >> A few months ago, we installed Cloudera Hadoop 3 in our local machine
> >> and everything was fine. Recently we also installed Whirr to start
> >> working with clusters. Although we faced some problems, after a while,
> >> we can start up a cluster, log into its master node and commence work.
> >> However, I found out recently that when I type:
> >>
> >> hadoop dfs -ls
> >>
> >> into our local machine, it now displays everything in the current
> >> directory I am in, not the contents of the DFS. This didn't use to
> >> happen, so we are thinking something got messed up when we installed
> >> Whirr.
> >>
> >> What could have caused this, and more importantly, how can we get our
> >> local hadoop dfs to point to the correct location?
> >>
> >> I posted this question on a Cloudera support board and I got the
> >> follow response:
> >>
> >> For your question, you should email the Whirr user list at
> >> whirr-user@incubator.apache.org. I suspect you have some old jar files
> >> on your classpath, but they'll know for sure.
> >>
> >> I appreciate the help and time.
> >>
> >> Thank you,
> >> Amy Zhang
> >
> >
>
Re: Why does my HDFS point to my current directory?
Posted by Amy Zhang <am...@gmail.com>.
Ok, it definitely looks like something got mixed up. Inside
/etc/hadoop/ I have conf, conf.empty, conf.pseudo, and conf.whirr.
The contents of conf are
[amyzhang@cactuar conf]$ ls
capacity-scheduler.xml hadoop-policy.xml ssl-client.xml.example
configuration.xsl log4j.properties ssl-server.xml.example
fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg
hadoop-env.sh masters whirr.log
hadoop-metrics.properties slaves whirr.log.2010-12-23
This is exactly the files I have in conf.whirr. So it seems my files
from conf.whirr replaced the original files in conf.
These are the files I have in conf.pseudo and conf.empty:
[amyzhang@cactuar conf.pseudo]$ ls
capacity-scheduler.xml hadoop-policy.xml slaves
configuration.xsl hdfs-site.xml ssl-client.xml.example
core-site.xml log4j.properties ssl-server.xml.example
fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg
hadoop-env.sh mapred-site.xml
hadoop-metrics.properties masters
What should I do at this point?
-Amy
On Mon, Jan 24, 2011 at 12:28 AM, Philip Zeyliger <ph...@cloudera.com> wrote:
> "hadoop dfs -ls" reads the configuration "fs.default.name" from somewhere,
> namely a copy of core-site.xml or hdfs-site.xml on your classpath.
> Typically, "hadoop dfs" looks in /etc/hadoop/conf, especially if you're
> using CDH. The default setting for fs.default.name is "file:///", which
> translates to your local file system. Perhaps you've lost that config file
> somewhere along the way?
> -- Philip
>
> On Sun, Jan 23, 2011 at 9:24 PM, Amy Zhang <am...@gmail.com> wrote:
>>
>> A few months ago, we installed Cloudera Hadoop 3 in our local machine
>> and everything was fine. Recently we also installed Whirr to start
>> working with clusters. Although we faced some problems, after a while,
>> we can start up a cluster, log into its master node and commence work.
>> However, I found out recently that when I type:
>>
>> hadoop dfs -ls
>>
>> into our local machine, it now displays everything in the current
>> directory I am in, not the contents of the DFS. This didn't use to
>> happen, so we are thinking something got messed up when we installed
>> Whirr.
>>
>> What could have caused this, and more importantly, how can we get our
>> local hadoop dfs to point to the correct location?
>>
>> I posted this question on a Cloudera support board and I got the
>> follow response:
>>
>> For your question, you should email the Whirr user list at
>> whirr-user@incubator.apache.org. I suspect you have some old jar files
>> on your classpath, but they'll know for sure.
>>
>> I appreciate the help and time.
>>
>> Thank you,
>> Amy Zhang
>
>
Re: Why does my HDFS point to my current directory?
Posted by Philip Zeyliger <ph...@cloudera.com>.
"hadoop dfs -ls" reads the configuration "fs.default.name" from somewhere,
namely a copy of core-site.xml or hdfs-site.xml on your classpath.
Typically, "hadoop dfs" looks in /etc/hadoop/conf, especially if you're
using CDH. The default setting for fs.default.name is "file:///", which
translates to your local file system. Perhaps you've lost that config file
somewhere along the way?
-- Philip
On Sun, Jan 23, 2011 at 9:24 PM, Amy Zhang <am...@gmail.com> wrote:
> A few months ago, we installed Cloudera Hadoop 3 in our local machine
> and everything was fine. Recently we also installed Whirr to start
> working with clusters. Although we faced some problems, after a while,
> we can start up a cluster, log into its master node and commence work.
> However, I found out recently that when I type:
>
> hadoop dfs -ls
>
> into our local machine, it now displays everything in the current
> directory I am in, not the contents of the DFS. This didn't use to
> happen, so we are thinking something got messed up when we installed
> Whirr.
>
> What could have caused this, and more importantly, how can we get our
> local hadoop dfs to point to the correct location?
>
> I posted this question on a Cloudera support board and I got the
> follow response:
>
> For your question, you should email the Whirr user list at
> whirr-user@incubator.apache.org. I suspect you have some old jar files
> on your classpath, but they'll know for sure.
>
> I appreciate the help and time.
>
> Thank you,
> Amy Zhang
>