You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Keith Wiley <kw...@keithwiley.com> on 2013/02/12 00:54:28 UTC
Whirr deployed hadoop cluster is very perplexing
I'm very confused by what I see when I use whirr to deploy a cluster. For example, the HDFS directory clearly mirrors the nonHDFS file system from the top dir, which is highly unconventional for hadoop, meaning that "$ ls /" shows the same thing as "$ hadoop fs -ls /":
$ hadoop fs -ls /
Found 25 items
drwxr-xr-x - root root 4096 2010-02-24 01:35 /bin
drwxr-xr-x - root root 4096 2010-02-24 01:40 /boot
drwxr-xr-x - root root 4096 2013-02-11 23:19 /data
drwxr-xr-x - root root 4096 2013-02-11 23:19 /data0
drwxr-xr-x - root root 12900 2013-02-11 23:14 /dev
drwxr-xr-x - root root 4096 2013-02-11 23:19 /etc
drwxr-xr-x - root root 4096 2013-02-11 23:15 /home
-rw-r--r-- 1 root root 6763173 2010-02-24 01:40 /initrd.img
-rw-r--r-- 1 root root 3689712 2010-02-24 01:36 /initrd.img.old
drwxr-xr-x - root root 12288 2010-02-24 01:40 /lib
drwx------ - root root 16384 2010-02-24 01:28 /lost+found
drwxr-xr-x - root root 4096 2010-02-24 01:31 /media
drwxr-xr-x - root root 4096 2013-02-11 23:19 /mnt
drwxr-xr-x - root root 4096 2010-02-24 01:31 /opt
dr-xr-xr-x - root root 0 2013-02-11 23:14 /proc
drwx------ - root root 4096 2013-02-11 23:14 /root
drwxr-xr-x - root root 4096 2010-02-24 01:40 /sbin
drwxr-xr-x - root root 4096 2009-12-05 21:55 /selinux
drwxr-xr-x - root root 4096 2010-02-24 01:31 /srv
drwxr-xr-x - root root 0 2013-02-11 23:14 /sys
drwxrwxrwt - root root 4096 2013-02-11 23:20 /tmp
drwxr-xr-x - root root 4096 2010-02-24 01:31 /usr
drwxr-xr-x - root root 4096 2010-02-24 01:36 /var
-rw-r--r-- 1 root root 3089086 2010-02-06 20:26 /vmlinuz
-rw-r--r-- 1 root root 4252096 2010-02-20 10:31 /vmlinuz.old
$
Likewise, if I create a directory outside HDFS, I then see it from within HDFS, so they really are looking at the same file system. That's not how HDFS is usually configured.
In addition, I can't actually operate within HDFS at all; I get an error as shown here:
$ hadoop fs -mkdir /testdir
mkdir: `/testdir': Input/output error
Even if I can straighten out these seemingly first-step issues, I also don't understand how to tell whirr to put HDFS on S3. I tried putting the following in hadoop.properties but I don't think it has any effect:
hadoop-hdfs.fs.default.name=s3://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY_esc}@somebucket
OR...
hadoop-hdfs.fs.default.name=s3://somebucket
hadoop-hdfs.fs.s3.awsAccessKeyId=${AWS_ACCESS_KEY_ID}
hadoop-hdfs.fs.s3.awsSecretAccessKey=${AWS_SECRET_ACCESS_KEY_esc}
I'm also not sure how to "su hadoop"; it asks for a password but I don't know what that would be. When I ssh in of course, it uses the account name from my computer (since that's the ssh command that whirr directly provides as it wraps up cluster deployment), but presumably to actually run a MapReduce job from the namenode I need to switch to the hadoop user, right (hmmm, is this why I couldn't create a directory within hadoop, as shown above)?
Incidentally, I also can't operate from my own machine because I can't get the proxy to connect either. It may have something to do with our corporate firewall, I'm not sure. For example, I get this:
$ export HADOOP_CONF_DIR=~/.whirr/hadoop-from-laptop/
$ hadoop fs -ls /
2013-02-11 15:34:07,767 WARN conf.Configuration (Configuration.java:<clinit>(477)) - DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
2013-02-11 15:34:08.337 java[8291:1203] Unable to load realm info from SCDynamicStore
2013-02-11 15:34:08.408 java[8291:1203] Unable to load realm info from SCDynamicStore
ls: Failed on local exception: java.net.SocketException: Malformed reply from SOCKS server; Host Details : local host is: "MyMachine.local/[ip-1]"; destination host is: "ec2-[ip-2].compute-1.amazonaws.com":8020;
~/ $
...while the proxy shell produces this error:
$ .whirr/hadoop-from-laptop/hadoop-proxy.sh
Running proxy to Hadoop cluster at ec2-54-234-185-62.compute-1.amazonaws.com. Use Ctrl-c to quit.
Warning: Permanently added '54.234.185.62' (RSA) to the list of known hosts.
channel 2: open failed: connect failed: Connection refused
Sooooooooo, I really don't understand what I'm seeing here: The HDFS directories don't like like a normal Hadoop cluster, they mirror the actual file system, I can't create directories within HDFS, I can't tell whirr to put HDFS on S3, and I can't use the proxy to interact with HDFS from my local machine. In fact, the ONLY thing I've managed to do so far is create the cluster in the first place.
This isn't working out very well so far. Where do I go from here?
Thanks.
________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com keithwiley.com music.keithwiley.com
"I used to be with it, but then they changed what it was. Now, what I'm with
isn't it, and what's it seems weird and scary to me."
-- Abe (Grandpa) Simpson
________________________________________________________________________________
Re: Whirr deployed hadoop cluster is very perplexing
Posted by Keith Wiley <kw...@keithwiley.com>.
I've been using Hadoop for years. I know about the various modes it can run in. The question is, if you think this problem arises from local mode, then why is Whirr apparently deploying in that mode in the first place. I pretty much followed the standard Whirr tutorials without modification, so why am I seeing behavior that differs from anyone else who would follow such a Whirr tutorial?
On Feb 11, 2013, at 20:40 , Mark Grover wrote:
> Hey Keith,
> I am pretty new to Whirr myself but I think what you are seeing is a configuration thing.
>
> What you are seeing is called local (or standalone) mode in Hadoop:
> http://hadoop.apache.org/docs/r0.20.2/quickstart.html#Local
>
> You will probably want to configure your cluster to be a psuedo-distributed cluster (if you are using one node) or regular distributed cluster (if you are using multiple nodes) for a closer to real-world scenario.
>
> Mark
________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com keithwiley.com music.keithwiley.com
"The easy confidence with which I know another man's religion is folly teaches
me to suspect that my own is also."
-- Mark Twain
________________________________________________________________________________
Re: Whirr deployed hadoop cluster is very perplexing
Posted by Keith Wiley <kw...@keithwiley.com>.
Where does Whirr set the standard Hadoop configurations and why would this be the default (meaning, I simply followed the common Whirr tutorials online verbatim so I got whatever the default behavior is).
On Feb 11, 2013, at 21:50 , Paul Baclace wrote:
> Check your fs.default.name setting in core-site.xml, it is probably set to file:/// ; the value of this property is prepended to a path that has no URI protocol when resolved by "hadoop fs -ls ..." and job inputs/outputs.
>
> This "feature" enables local mode (single process) and pseudo or full cluster modes to work using the same absolute paths without protocol:// in test scripts. I think I used it once in 6 years, but someone out there might rely on it.
>
> The above means that when using a real cluster,
>
> hadoop fs -ls file:///
>
> will indeed show the local root filesystem.
>
>
> Paul
________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com keithwiley.com music.keithwiley.com
"What I primarily learned in grad school is how much I *don't* know.
Consequently, I left grad school with a higher ignorance to knowledge ratio than
when I entered."
-- Keith Wiley
________________________________________________________________________________
Re: Whirr deployed hadoop cluster is very perplexing
Posted by Paul Baclace <pa...@gmail.com>.
Check your fs.default.name setting in core-site.xml, it is probably set
to file:/// ; the value of this property is prepended to a path that has
no URI protocol when resolved by "hadoop fs -ls ..." and job
inputs/outputs.
This "feature" enables local mode (single process) and pseudo or full
cluster modes to work using the same absolute paths without protocol://
in test scripts. I think I used it once in 6 years, but someone out
there might rely on it.
The above means that when using a real cluster,
hadoop fs -ls file:///
will indeed show the local root filesystem.
Paul
On 20130211 20:40 , Mark Grover wrote:
> Hey Keith,
> I am pretty new to Whirr myself but I think what you are seeing is a
> configuration thing.
>
> What you are seeing is called local (or standalone) mode in Hadoop:
> http://hadoop.apache.org/docs/r0.20.2/quickstart.html#Local
>
> You will probably want to configure your cluster to be a
> psuedo-distributed cluster (if you are using one node) or regular
> distributed cluster (if you are using multiple nodes) for a closer to
> real-world scenario.
>
> Mark
>
> On Mon, Feb 11, 2013 at 3:54 PM, Keith Wiley <kwiley@keithwiley.com
> <ma...@keithwiley.com>> wrote:
>
> I'm very confused by what I see when I use whirr to deploy a
> cluster. For example, the HDFS directory clearly mirrors the
> nonHDFS file system from the top dir, which is highly
> unconventional for hadoop, meaning that "$ ls /" shows the same
> thing as "$ hadoop fs -ls /":
>
> $ hadoop fs -ls /
> Found 25 items
> drwxr-xr-x - root root 4096 2010-02-24 01:35 /bin
> drwxr-xr-x - root root 4096 2010-02-24 01:40 /boot
> drwxr-xr-x - root root 4096 2013-02-11 23:19 /data
> drwxr-xr-x - root root 4096 2013-02-11 23:19 /data0
> drwxr-xr-x - root root 12900 2013-02-11 23:14 /dev
> drwxr-xr-x - root root 4096 2013-02-11 23:19 /etc
> drwxr-xr-x - root root 4096 2013-02-11 23:15 /home
> -rw-r--r-- 1 root root 6763173 2010-02-24 01:40 /initrd.img
> -rw-r--r-- 1 root root 3689712 2010-02-24 01:36 /initrd.img.old
> drwxr-xr-x - root root 12288 2010-02-24 01:40 /lib
> drwx------ - root root 16384 2010-02-24 01:28 /lost+found
> drwxr-xr-x - root root 4096 2010-02-24 01:31 /media
> drwxr-xr-x - root root 4096 2013-02-11 23:19 /mnt
> drwxr-xr-x - root root 4096 2010-02-24 01:31 /opt
> dr-xr-xr-x - root root 0 2013-02-11 23:14 /proc
> drwx------ - root root 4096 2013-02-11 23:14 /root
> drwxr-xr-x - root root 4096 2010-02-24 01:40 /sbin
> drwxr-xr-x - root root 4096 2009-12-05 21:55 /selinux
> drwxr-xr-x - root root 4096 2010-02-24 01:31 /srv
> drwxr-xr-x - root root 0 2013-02-11 23:14 /sys
> drwxrwxrwt - root root 4096 2013-02-11 23:20 /tmp
> drwxr-xr-x - root root 4096 2010-02-24 01:31 /usr
> drwxr-xr-x - root root 4096 2010-02-24 01:36 /var
> -rw-r--r-- 1 root root 3089086 2010-02-06 20:26 /vmlinuz
> -rw-r--r-- 1 root root 4252096 2010-02-20 10:31 /vmlinuz.old
> $
>
> Likewise, if I create a directory outside HDFS, I then see it from
> within HDFS, so they really are looking at the same file system.
> That's not how HDFS is usually configured.
>
> In addition, I can't actually operate within HDFS at all; I get an
> error as shown here:
> $ hadoop fs -mkdir /testdir
> mkdir: `/testdir': Input/output error
>
> Even if I can straighten out these seemingly first-step issues, I
> also don't understand how to tell whirr to put HDFS on S3. I
> tried putting the following in hadoop.properties but I don't think
> it has any effect:
>
> hadoop-hdfs.fs.default.name
> <http://hadoop-hdfs.fs.default.name>=s3://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY_esc}@somebucket
> OR...
> hadoop-hdfs.fs.default.name
> <http://hadoop-hdfs.fs.default.name>=s3://somebucket
> hadoop-hdfs.fs.s3.awsAccessKeyId=${AWS_ACCESS_KEY_ID}
> hadoop-hdfs.fs.s3.awsSecretAccessKey=${AWS_SECRET_ACCESS_KEY_esc}
>
> I'm also not sure how to "su hadoop"; it asks for a password but I
> don't know what that would be. When I ssh in of course, it uses
> the account name from my computer (since that's the ssh command
> that whirr directly provides as it wraps up cluster deployment),
> but presumably to actually run a MapReduce job from the namenode I
> need to switch to the hadoop user, right (hmmm, is this why I
> couldn't create a directory within hadoop, as shown above)?
>
> Incidentally, I also can't operate from my own machine because I
> can't get the proxy to connect either. It may have something to
> do with our corporate firewall, I'm not sure. For example, I get
> this:
>
> $ export HADOOP_CONF_DIR=~/.whirr/hadoop-from-laptop/
> $ hadoop fs -ls /
> 2013-02-11 15:34:07,767 WARN conf.Configuration
> (Configuration.java:<clinit>(477)) - DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated.
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
> override properties of core-default.xml, mapred-default.xml and
> hdfs-default.xml respectively
> 2013-02-11 15:34:08.337 java[8291:1203] Unable to load realm info
> from SCDynamicStore
> 2013-02-11 15:34:08.408 java[8291:1203] Unable to load realm info
> from SCDynamicStore
> ls: Failed on local exception: java.net.SocketException: Malformed
> reply from SOCKS server; Host Details : local host is:
> "MyMachine.local/[ip-1]"; destination host is:
> "ec2-[ip-2].compute-1.amazonaws.com
> <http://compute-1.amazonaws.com>":8020;
> ~/ $
> ...while the proxy shell produces this error:
> $ .whirr/hadoop-from-laptop/hadoop-proxy.sh
> Running proxy to Hadoop cluster at
> ec2-54-234-185-62.compute-1.amazonaws.com
> <http://ec2-54-234-185-62.compute-1.amazonaws.com>. Use Ctrl-c to
> quit.
> Warning: Permanently added '54.234.185.62' (RSA) to the list of
> known hosts.
> channel 2: open failed: connect failed: Connection refused
>
> Sooooooooo, I really don't understand what I'm seeing here: The
> HDFS directories don't like like a normal Hadoop cluster, they
> mirror the actual file system, I can't create directories within
> HDFS, I can't tell whirr to put HDFS on S3, and I can't use the
> proxy to interact with HDFS from my local machine. In fact, the
> ONLY thing I've managed to do so far is create the cluster in the
> first place.
>
> This isn't working out very well so far. Where do I go from here?
>
> Thanks.
>
>
> ________________________________________________________________________________
> Keith Wiley kwiley@keithwiley.com <ma...@keithwiley.com>
> keithwiley.com <http://keithwiley.com> music.keithwiley.com
> <http://music.keithwiley.com>
>
> "I used to be with it, but then they changed what it was. Now,
> what I'm with
> isn't it, and what's it seems weird and scary to me."
> -- Abe (Grandpa) Simpson
> ________________________________________________________________________________
>
>
Re: Whirr deployed hadoop cluster is very perplexing
Posted by Mark Grover <gr...@gmail.com>.
Hey Keith,
I am pretty new to Whirr myself but I think what you are seeing is a
configuration thing.
What you are seeing is called local (or standalone) mode in Hadoop:
http://hadoop.apache.org/docs/r0.20.2/quickstart.html#Local
You will probably want to configure your cluster to be a psuedo-distributed
cluster (if you are using one node) or regular distributed cluster (if you
are using multiple nodes) for a closer to real-world scenario.
Mark
On Mon, Feb 11, 2013 at 3:54 PM, Keith Wiley <kw...@keithwiley.com> wrote:
> I'm very confused by what I see when I use whirr to deploy a cluster. For
> example, the HDFS directory clearly mirrors the nonHDFS file system from
> the top dir, which is highly unconventional for hadoop, meaning that "$ ls
> /" shows the same thing as "$ hadoop fs -ls /":
>
> $ hadoop fs -ls /
> Found 25 items
> drwxr-xr-x - root root 4096 2010-02-24 01:35 /bin
> drwxr-xr-x - root root 4096 2010-02-24 01:40 /boot
> drwxr-xr-x - root root 4096 2013-02-11 23:19 /data
> drwxr-xr-x - root root 4096 2013-02-11 23:19 /data0
> drwxr-xr-x - root root 12900 2013-02-11 23:14 /dev
> drwxr-xr-x - root root 4096 2013-02-11 23:19 /etc
> drwxr-xr-x - root root 4096 2013-02-11 23:15 /home
> -rw-r--r-- 1 root root 6763173 2010-02-24 01:40 /initrd.img
> -rw-r--r-- 1 root root 3689712 2010-02-24 01:36 /initrd.img.old
> drwxr-xr-x - root root 12288 2010-02-24 01:40 /lib
> drwx------ - root root 16384 2010-02-24 01:28 /lost+found
> drwxr-xr-x - root root 4096 2010-02-24 01:31 /media
> drwxr-xr-x - root root 4096 2013-02-11 23:19 /mnt
> drwxr-xr-x - root root 4096 2010-02-24 01:31 /opt
> dr-xr-xr-x - root root 0 2013-02-11 23:14 /proc
> drwx------ - root root 4096 2013-02-11 23:14 /root
> drwxr-xr-x - root root 4096 2010-02-24 01:40 /sbin
> drwxr-xr-x - root root 4096 2009-12-05 21:55 /selinux
> drwxr-xr-x - root root 4096 2010-02-24 01:31 /srv
> drwxr-xr-x - root root 0 2013-02-11 23:14 /sys
> drwxrwxrwt - root root 4096 2013-02-11 23:20 /tmp
> drwxr-xr-x - root root 4096 2010-02-24 01:31 /usr
> drwxr-xr-x - root root 4096 2010-02-24 01:36 /var
> -rw-r--r-- 1 root root 3089086 2010-02-06 20:26 /vmlinuz
> -rw-r--r-- 1 root root 4252096 2010-02-20 10:31 /vmlinuz.old
> $
>
> Likewise, if I create a directory outside HDFS, I then see it from within
> HDFS, so they really are looking at the same file system. That's not how
> HDFS is usually configured.
>
> In addition, I can't actually operate within HDFS at all; I get an error
> as shown here:
> $ hadoop fs -mkdir /testdir
> mkdir: `/testdir': Input/output error
>
> Even if I can straighten out these seemingly first-step issues, I also
> don't understand how to tell whirr to put HDFS on S3. I tried putting the
> following in hadoop.properties but I don't think it has any effect:
>
> hadoop-hdfs.fs.default.name
> =s3://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY_esc}@somebucket
> OR...
> hadoop-hdfs.fs.default.name=s3://somebucket
> hadoop-hdfs.fs.s3.awsAccessKeyId=${AWS_ACCESS_KEY_ID}
> hadoop-hdfs.fs.s3.awsSecretAccessKey=${AWS_SECRET_ACCESS_KEY_esc}
>
> I'm also not sure how to "su hadoop"; it asks for a password but I don't
> know what that would be. When I ssh in of course, it uses the account name
> from my computer (since that's the ssh command that whirr directly provides
> as it wraps up cluster deployment), but presumably to actually run a
> MapReduce job from the namenode I need to switch to the hadoop user, right
> (hmmm, is this why I couldn't create a directory within hadoop, as shown
> above)?
>
> Incidentally, I also can't operate from my own machine because I can't get
> the proxy to connect either. It may have something to do with our
> corporate firewall, I'm not sure. For example, I get this:
>
> $ export HADOOP_CONF_DIR=~/.whirr/hadoop-from-laptop/
> $ hadoop fs -ls /
> 2013-02-11 15:34:07,767 WARN conf.Configuration
> (Configuration.java:<clinit>(477)) - DEPRECATED: hadoop-site.xml found in
> the classpath. Usage of hadoop-site.xml is deprecated. Instead use
> core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
> core-default.xml, mapred-default.xml and hdfs-default.xml respectively
> 2013-02-11 15:34:08.337 java[8291:1203] Unable to load realm info from
> SCDynamicStore
> 2013-02-11 15:34:08.408 java[8291:1203] Unable to load realm info from
> SCDynamicStore
> ls: Failed on local exception: java.net.SocketException: Malformed reply
> from SOCKS server; Host Details : local host is: "MyMachine.local/[ip-1]";
> destination host is: "ec2-[ip-2].compute-1.amazonaws.com":8020;
> ~/ $
> ...while the proxy shell produces this error:
> $ .whirr/hadoop-from-laptop/hadoop-proxy.sh
> Running proxy to Hadoop cluster at
> ec2-54-234-185-62.compute-1.amazonaws.com. Use Ctrl-c to quit.
> Warning: Permanently added '54.234.185.62' (RSA) to the list of known
> hosts.
> channel 2: open failed: connect failed: Connection refused
>
> Sooooooooo, I really don't understand what I'm seeing here: The HDFS
> directories don't like like a normal Hadoop cluster, they mirror the actual
> file system, I can't create directories within HDFS, I can't tell whirr to
> put HDFS on S3, and I can't use the proxy to interact with HDFS from my
> local machine. In fact, the ONLY thing I've managed to do so far is create
> the cluster in the first place.
>
> This isn't working out very well so far. Where do I go from here?
>
> Thanks.
>
>
>
> ________________________________________________________________________________
> Keith Wiley kwiley@keithwiley.com keithwiley.com
> music.keithwiley.com
>
> "I used to be with it, but then they changed what it was. Now, what I'm
> with
> isn't it, and what's it seems weird and scary to me."
> -- Abe (Grandpa) Simpson
>
> ________________________________________________________________________________
>
>