You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Toby DiPasquale <co...@gmail.com> on 2007/09/05 21:19:59 UTC
Accessing S3 with Hadoop?
Hi all,
I'm trying to get Hadoop 0.14.0 to talk to S3 but its not working out
for me. My credentials are definitely right (checked multiple times
and that's not the issue in any case) and the bucket I'm querying
definitely does exist already and has keys under it.
However, when I try to access S3, I get really weird stuff going on.
If I try this:
bin/hadoop fs -ls key-with-stuff-under-it
it fails. I set it to DEBUG and I can see that the S3 library is
trying to query:
GET /BUCKET/%2Fuser%2Froot%2Fkey-with-stuff-under-it
which is wrong for obvious reasons, but which I can sort of
understand, as the default value of the workingDir instance variable
in S3FileSystem.java is "/user/${user.name}" (anybody know where this
default comes from, btw? it make no sense to me as is).
When I try the full URI:
bin/hadoop fs -ls s3://BUCKET/key-with-stuff-under-it
it still fails, instead doing this even stranger thing and trying to query
GET /BUCKET/%2Fkey-with-stuff-under-it
Note: the embedded %2F in there that makes it incorrect.
It also does this same thing whether I embed the credentials in the
full s3:// URI or not.
Then, I thought maybe it doesn't want a bucket at all in
fs.default.name, so I took that out and then the configuration failed
to parse on startup, throwing an IllegalArgumentException, telling me
it was expecting an authority portion to the URI it was given.
Are the docs on the wiki and Amazon's articles wrong about how to
configure hadoop-site.xml for S3 as a filesystem? Or am I doing
something wrong in my config, leaving something out, etc? Any
thoughts? Should I be setting the workingDir somewhere in the config
to something more sensible? I couldn't find much about this on the
mailing list or the forums at Amazon so I thought I'd just ask. My
config info is below.
Any help is greatly appreciated. Thanks :)
My mapred-default.xml is the default one from
hadoop-0.14.0/src/contrib/ec2/image/hadoop-init. I am running this on
EC2 with just MapReduce (no HDFS NameNodes/DataNodes running). Here's
my hadoop-site.xml (anonymized, of course):
------------------------------------
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/mnt/hadoop</value>
</property>
<property>
<name>fs.default.name</name>
<value>s3://BUCKET</value>
</property>
<property>
<name>fs.s3.awsAccessKeyId</name>
<value>XXXXXXXXXXXXXXXXXX</value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>XXXXXXXXXXXXXXXXXX</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>$MASTER_HOST:50002</value>
</property>
</configuration>
------------------------------------
--
Toby DiPasquale
Re: Accessing S3 with Hadoop?
Posted by Toby DiPasquale <co...@gmail.com>.
On 9/13/07, Tom White <to...@gmail.com> wrote:
> > > Well, no, since my $MASTER_HOST is now just the external DNS name of
> > > the first instance started in the reservation, but this is performed
> > > as part of my launch-hadoop-cluster script. In any case, that value is
> > > not set to the internal IP, but rather to the hostname portion of the
> > > internal DNS name.
> >
> > This is a bit of a mystery to me - I'll try to reproduce it in on my
> > workstation.
> >
>
> I tried running the EC2 scripts and they work fine for me on Linux and
> OSX. Do you have the same problems you report without the
> modifications you made to the scripts (apart from those to
> hadoop-ec2-env.sh)? How about if you use HDFS rather than S3?
I don't remember checking until I had modified the scripts since it
wasn't working well enough to have to worry about that ;-) I haven't
tried HDFS although I plan to do so this weekend.
> I've also created a public 0.14.1 AMI so that might be worth trying too.
The code in src/contrib/ec2 is identical between 0.14.0 and 0.14.1 so
I don't see what could be different. I was going to try and move to
0.14.1 but then I diff'd the two and found that they were the same so
I was going to work out this issue before I moved up.
--
Toby DiPasquale
Re: Accessing S3 with Hadoop?
Posted by Tom White <to...@gmail.com>.
Hi Toby,
> > There is a patch for HADOOP-1783 (see the Jira attachment) which fixes
> > the issue - the one-liner didn't fix unit tests. This didn't make the
> > 0.14.1 release but it will make the next one.
>
> As I said in the previous email, this patch does not apply to 0.14.0 or 0.14.1:
>
> black:~/tmp/hadoop-0.14.1> patch -p0 < ~/Downloads/HADOOP-1783.patch
> patching file src/test/org/apache/hadoop/fs/s3/InMemoryFileSystemStore.java
> patching file src/test/org/apache/hadoop/fs/s3/S3FileSystemBaseTest.java
> patching file src/java/org/apache/hadoop/fs/s3/S3FileSystem.java
> Hunk #1 FAILED at 173.
> 1 out of 1 hunk FAILED -- saving rejects to file
> src/java/org/apache/hadoop/fs/s3/S3FileSystem.java.rej
> black:~/tmp/hadoop-0.14.1>
>
> This is why I had to use the one-liner from the comment on that bug
> rather than the attached patch. Any ideas?
Only just round to looking at this - I've now fixed this in trunk and
in the 0.14 branch. The patch didn't apply to the 0.14 branch since it
relied on a change in trunk to do with FileStatus.
Cheers,
Tom
Re: Accessing S3 with Hadoop?
Posted by Toby DiPasquale <co...@gmail.com>.
On 9/17/07, Tom White <to...@gmail.com> wrote:
> I didn't realize that you were trying to use your own AMI! The
> procedure on the wiki for building your own is based on using the core
> Fedora AMI. I did try using an Ubuntu one some time back but got into
> difficulties with networking, so I've used the official Fedora one
> since then. Anyway, glad you've got something working.
Sorry, I should have mentioned that earlier, although I am running the
scripts from off-EC2 (my OS X laptop, to be exact). Personally, I
prefer not to use any RedHat-based distros, as making RPMs is a major
PITA compared to making a dpkg.
> There is a patch for HADOOP-1783 (see the Jira attachment) which fixes
> the issue - the one-liner didn't fix unit tests. This didn't make the
> 0.14.1 release but it will make the next one.
As I said in the previous email, this patch does not apply to 0.14.0 or 0.14.1:
black:~/tmp/hadoop-0.14.1> patch -p0 < ~/Downloads/HADOOP-1783.patch
patching file src/test/org/apache/hadoop/fs/s3/InMemoryFileSystemStore.java
patching file src/test/org/apache/hadoop/fs/s3/S3FileSystemBaseTest.java
patching file src/java/org/apache/hadoop/fs/s3/S3FileSystem.java
Hunk #1 FAILED at 173.
1 out of 1 hunk FAILED -- saving rejects to file
src/java/org/apache/hadoop/fs/s3/S3FileSystem.java.rej
black:~/tmp/hadoop-0.14.1>
This is why I had to use the one-liner from the comment on that bug
rather than the attached patch. Any ideas?
--
Toby DiPasquale
Re: Accessing S3 with Hadoop?
Posted by Tom White <to...@gmail.com>.
> The issue for me regarding the hostname thing was that my Ubuntu-based
> AMI was not setting the hostname correctly to the external DNS name
> (my fault, not Ubuntu's). I've fixed that and so the canonical scripts
> from hadoop-0.14.[01]/src/contrib/ec2 should work for me now, though I
> am still using my own for the time being.
I didn't realize that you were trying to use your own AMI! The
procedure on the wiki for building your own is based on using the core
Fedora AMI. I did try using an Ubuntu one some time back but got into
difficulties with networking, so I've used the official Fedora one
since then. Anyway, glad you've got something working.
> I've also moved up to Hadoop 0.14.1 and hand-applied the fix from
> HADOOP-1783. I'm referring to the the one-liner in the second comment;
> the patch attached to the bug doesn't apply to 0.14.0 or 0.14.1. Was
> there any reason why the fix from the comment didn't make it into
> 0.14.1?
There is a patch for HADOOP-1783 (see the Jira attachment) which fixes
the issue - the one-liner didn't fix unit tests. This didn't make the
0.14.1 release but it will make the next one.
Cheers,
Tom
Re: Accessing S3 with Hadoop?
Posted by Toby DiPasquale <co...@gmail.com>.
On 9/13/07, Tom White <to...@gmail.com> wrote:
> I tried running the EC2 scripts and they work fine for me on Linux and
> OSX. Do you have the same problems you report without the
> modifications you made to the scripts (apart from those to
> hadoop-ec2-env.sh)? How about if you use HDFS rather than S3?
>
> I've also created a public 0.14.1 AMI so that might be worth trying too.
The issue for me regarding the hostname thing was that my Ubuntu-based
AMI was not setting the hostname correctly to the external DNS name
(my fault, not Ubuntu's). I've fixed that and so the canonical scripts
from hadoop-0.14.[01]/src/contrib/ec2 should work for me now, though I
am still using my own for the time being.
I've also moved up to Hadoop 0.14.1 and hand-applied the fix from
HADOOP-1783. I'm referring to the the one-liner in the second comment;
the patch attached to the bug doesn't apply to 0.14.0 or 0.14.1. Was
there any reason why the fix from the comment didn't make it into
0.14.1?
--
Toby DiPasquale
Re: Accessing S3 with Hadoop?
Posted by Tom White <to...@gmail.com>.
Hi Toby,
> > > Are you able to do 'bin/hadoop-ec2 launch-cluster' then (on your workstation)
> > >
> > > . bin/hadoop-ec2-env.sh
> > > ssh $SSH_OPTS "root@$MASTER_HOST" "sed -i -e
> > > \"s/$MASTER_HOST/\$(hostname)/g\"
> > > /usr/local/hadoop-$HADOOP_VERSION/conf/hadoop-site.xml"
> > >
> > > and then check to see if the master host has been set correctly (to
> > > the internal IP) in the master host's hadoop-site.xml.
> >
> > Well, no, since my $MASTER_HOST is now just the external DNS name of
> > the first instance started in the reservation, but this is performed
> > as part of my launch-hadoop-cluster script. In any case, that value is
> > not set to the internal IP, but rather to the hostname portion of the
> > internal DNS name.
>
> This is a bit of a mystery to me - I'll try to reproduce it in on my
> workstation.
>
I tried running the EC2 scripts and they work fine for me on Linux and
OSX. Do you have the same problems you report without the
modifications you made to the scripts (apart from those to
hadoop-ec2-env.sh)? How about if you use HDFS rather than S3?
I've also created a public 0.14.1 AMI so that might be worth trying too.
Tom
Re: hadoop client
Posted by Allen Wittenauer <aw...@yahoo-inc.com>.
On 9/11/07 2:23 PM, "Earney, Billy C." <ea...@umsystem.edu> wrote:
> So I guess you can run the command below ($HADOOP_HOME/bin/hadoop ..) on
> a separate machine just while you have the config file which defines the
> ip address of the namenode and/or the datanodes?
Essentially, yes.
We routinely have nodes that aren't part of the DFS load data onto the
DFS. [In fact, in one of our larger configurations, we have ~100 nodes that
aren't datanodes but are used for mapred operations against the dfs..
primarily because we've hit some namenode limits with 0.13. :) ] The key
thing is to have the config bits available to pass to the hadoop command.
As I mentioned in an email previously, we routinely do NOT do put's from
datanodes because they get 'priority' for block placement. If you are doing
a large file put, this means you fill your local node. This may not be
ideal depending upon your requirements.
RE: hadoop client
Posted by "Earney, Billy C." <ea...@umsystem.edu>.
So I guess you can run the command below ($HADOOP_HOME/bin/hadoop ..) on
a separate machine just while you have the config file which defines the
ip address of the namenode and/or the datanodes?
Thanks!
-----Original Message-----
From: Michael Bieniosek [mailto:michael@powerset.com]
Sent: Tuesday, September 11, 2007 3:54 PM
To: hadoop-user@lucene.apache.org; Earney, Billy C.
Subject: Re: hadoop client
There is a java hadoop client you can run with
$HADOOP_HOME/bin/hadoop [--config /path/to/config/dir] fs -help
Supposedly there are also webdav and fuse HDFS implementations, but I
don't
know anything about them.
-Michael
On 9/11/07 1:11 PM, "Earney, Billy C." <ea...@umsystem.edu> wrote:
> Greetings!
>
> I've been reading through the documentation, and there is one piece of
> information I'm not finding (or I missed..). Lets say you have a
> cluster of machines one being the namenode, and the rest serving as
> datanodes. Does a client process (a process trying to
> insert/delete/read files) need to be running on the namenode or
> datanodes? (or can it run on another machine?).
>
> If a client process can run on another machine, can someone give an
> example and the configuration to do such a thing? I've seen there has
> been some work done on webdav with hadoop, and was wondering if a
> machine not part of cluster could access HDFS with something like
webdav
> (or similar tool)?
>
> Thanks!
>
> -----Original Message-----
> From: Tom White [mailto:tom.e.white@gmail.com]
> Sent: Tuesday, September 11, 2007 2:16 PM
> To: hadoop-user@lucene.apache.org
> Subject: Re: Accessing S3 with Hadoop?
>
>> I just updated the page to add a Notes section explaining the issue
>> and referencing the JIRA issue # you mentioned earlier.
>
> Great - thanks.
>
>>> Are you able to do 'bin/hadoop-ec2 launch-cluster' then (on your
> workstation)
>>>
>>> . bin/hadoop-ec2-env.sh
>>> ssh $SSH_OPTS "root@$MASTER_HOST" "sed -i -e
>>> \"s/$MASTER_HOST/\$(hostname)/g\"
>>> /usr/local/hadoop-$HADOOP_VERSION/conf/hadoop-site.xml"
>>>
>>> and then check to see if the master host has been set correctly (to
>>> the internal IP) in the master host's hadoop-site.xml.
>>
>> Well, no, since my $MASTER_HOST is now just the external DNS name of
>> the first instance started in the reservation, but this is performed
>> as part of my launch-hadoop-cluster script. In any case, that value
is
>> not set to the internal IP, but rather to the hostname portion of the
>> internal DNS name.
>
> This is a bit of a mystery to me - I'll try to reproduce it in on my
> workstation.
>
>>
>> Currently, my MR jobs are failing because the reducers can't copy the
>> map output and I'm thinking it might be because there is some kind of
>> external address getting in there somehow. I see connections to
>> external IPs in netstat -tan (72.* addresses). Any ideas about that?
>> In the hadoop-site.xml's on the slaves, the address is the external
>> DNS name of the master (ec2-*) but that resolves to the internal 10/8
>> address like it should.
>>
>>> Also, what version of the EC2 tools are you using?
>>
>> black:~/code/hadoop-0.14.0/src/contrib/ec2> ec2-version
>> 1.2-11797 2007-03-01
>> black:~/code/hadoop-0.14.0/src/contrib/ec2>
>
> I'm using the same version so that's not it.
>
>>> Instances are terminated on the basis of their AMI ID since 0.14.0.
>>> See https://issues.apache.org/jira/browse/HADOOP-1504.
>>
>> I felt this was unsafe as it was, since it looked for a name of an
>> image and then reversed it to the AMI ID. I just hacked it so you
have
>> to put in the AMI ID in hadoop-ec2-env.sh. Also, the script as it is
>> right now doesn't grep for 'running' so may potentially shut down
some
>> instances starting up in another cluster. I may just be paranoid,
>> however ;)
>
> Checking for 'running' is a good idea. I've relied on version number
> so folks can easily select the version of hadoop they want on the
> cluster. Perhaps the best solution would be to allow an optional
> parameter to the terminate script to specify the AMI ID if you need
> extra certainty (the script already prompts with a list of instances
> to terminate).
>
> Tom
Re: hadoop client
Posted by Michael Bieniosek <mi...@powerset.com>.
There is a java hadoop client you can run with
$HADOOP_HOME/bin/hadoop [--config /path/to/config/dir] fs -help
Supposedly there are also webdav and fuse HDFS implementations, but I don't
know anything about them.
-Michael
On 9/11/07 1:11 PM, "Earney, Billy C." <ea...@umsystem.edu> wrote:
> Greetings!
>
> I've been reading through the documentation, and there is one piece of
> information I'm not finding (or I missed..). Lets say you have a
> cluster of machines one being the namenode, and the rest serving as
> datanodes. Does a client process (a process trying to
> insert/delete/read files) need to be running on the namenode or
> datanodes? (or can it run on another machine?).
>
> If a client process can run on another machine, can someone give an
> example and the configuration to do such a thing? I've seen there has
> been some work done on webdav with hadoop, and was wondering if a
> machine not part of cluster could access HDFS with something like webdav
> (or similar tool)?
>
> Thanks!
>
> -----Original Message-----
> From: Tom White [mailto:tom.e.white@gmail.com]
> Sent: Tuesday, September 11, 2007 2:16 PM
> To: hadoop-user@lucene.apache.org
> Subject: Re: Accessing S3 with Hadoop?
>
>> I just updated the page to add a Notes section explaining the issue
>> and referencing the JIRA issue # you mentioned earlier.
>
> Great - thanks.
>
>>> Are you able to do 'bin/hadoop-ec2 launch-cluster' then (on your
> workstation)
>>>
>>> . bin/hadoop-ec2-env.sh
>>> ssh $SSH_OPTS "root@$MASTER_HOST" "sed -i -e
>>> \"s/$MASTER_HOST/\$(hostname)/g\"
>>> /usr/local/hadoop-$HADOOP_VERSION/conf/hadoop-site.xml"
>>>
>>> and then check to see if the master host has been set correctly (to
>>> the internal IP) in the master host's hadoop-site.xml.
>>
>> Well, no, since my $MASTER_HOST is now just the external DNS name of
>> the first instance started in the reservation, but this is performed
>> as part of my launch-hadoop-cluster script. In any case, that value is
>> not set to the internal IP, but rather to the hostname portion of the
>> internal DNS name.
>
> This is a bit of a mystery to me - I'll try to reproduce it in on my
> workstation.
>
>>
>> Currently, my MR jobs are failing because the reducers can't copy the
>> map output and I'm thinking it might be because there is some kind of
>> external address getting in there somehow. I see connections to
>> external IPs in netstat -tan (72.* addresses). Any ideas about that?
>> In the hadoop-site.xml's on the slaves, the address is the external
>> DNS name of the master (ec2-*) but that resolves to the internal 10/8
>> address like it should.
>>
>>> Also, what version of the EC2 tools are you using?
>>
>> black:~/code/hadoop-0.14.0/src/contrib/ec2> ec2-version
>> 1.2-11797 2007-03-01
>> black:~/code/hadoop-0.14.0/src/contrib/ec2>
>
> I'm using the same version so that's not it.
>
>>> Instances are terminated on the basis of their AMI ID since 0.14.0.
>>> See https://issues.apache.org/jira/browse/HADOOP-1504.
>>
>> I felt this was unsafe as it was, since it looked for a name of an
>> image and then reversed it to the AMI ID. I just hacked it so you have
>> to put in the AMI ID in hadoop-ec2-env.sh. Also, the script as it is
>> right now doesn't grep for 'running' so may potentially shut down some
>> instances starting up in another cluster. I may just be paranoid,
>> however ;)
>
> Checking for 'running' is a good idea. I've relied on version number
> so folks can easily select the version of hadoop they want on the
> cluster. Perhaps the best solution would be to allow an optional
> parameter to the terminate script to specify the AMI ID if you need
> extra certainty (the script already prompts with a list of instances
> to terminate).
>
> Tom
hadoop client
Posted by "Earney, Billy C." <ea...@umsystem.edu>.
Greetings!
I've been reading through the documentation, and there is one piece of
information I'm not finding (or I missed..). Lets say you have a
cluster of machines one being the namenode, and the rest serving as
datanodes. Does a client process (a process trying to
insert/delete/read files) need to be running on the namenode or
datanodes? (or can it run on another machine?).
If a client process can run on another machine, can someone give an
example and the configuration to do such a thing? I've seen there has
been some work done on webdav with hadoop, and was wondering if a
machine not part of cluster could access HDFS with something like webdav
(or similar tool)?
Thanks!
-----Original Message-----
From: Tom White [mailto:tom.e.white@gmail.com]
Sent: Tuesday, September 11, 2007 2:16 PM
To: hadoop-user@lucene.apache.org
Subject: Re: Accessing S3 with Hadoop?
> I just updated the page to add a Notes section explaining the issue
> and referencing the JIRA issue # you mentioned earlier.
Great - thanks.
> > Are you able to do 'bin/hadoop-ec2 launch-cluster' then (on your
workstation)
> >
> > . bin/hadoop-ec2-env.sh
> > ssh $SSH_OPTS "root@$MASTER_HOST" "sed -i -e
> > \"s/$MASTER_HOST/\$(hostname)/g\"
> > /usr/local/hadoop-$HADOOP_VERSION/conf/hadoop-site.xml"
> >
> > and then check to see if the master host has been set correctly (to
> > the internal IP) in the master host's hadoop-site.xml.
>
> Well, no, since my $MASTER_HOST is now just the external DNS name of
> the first instance started in the reservation, but this is performed
> as part of my launch-hadoop-cluster script. In any case, that value is
> not set to the internal IP, but rather to the hostname portion of the
> internal DNS name.
This is a bit of a mystery to me - I'll try to reproduce it in on my
workstation.
>
> Currently, my MR jobs are failing because the reducers can't copy the
> map output and I'm thinking it might be because there is some kind of
> external address getting in there somehow. I see connections to
> external IPs in netstat -tan (72.* addresses). Any ideas about that?
> In the hadoop-site.xml's on the slaves, the address is the external
> DNS name of the master (ec2-*) but that resolves to the internal 10/8
> address like it should.
>
> > Also, what version of the EC2 tools are you using?
>
> black:~/code/hadoop-0.14.0/src/contrib/ec2> ec2-version
> 1.2-11797 2007-03-01
> black:~/code/hadoop-0.14.0/src/contrib/ec2>
I'm using the same version so that's not it.
> > Instances are terminated on the basis of their AMI ID since 0.14.0.
> > See https://issues.apache.org/jira/browse/HADOOP-1504.
>
> I felt this was unsafe as it was, since it looked for a name of an
> image and then reversed it to the AMI ID. I just hacked it so you have
> to put in the AMI ID in hadoop-ec2-env.sh. Also, the script as it is
> right now doesn't grep for 'running' so may potentially shut down some
> instances starting up in another cluster. I may just be paranoid,
> however ;)
Checking for 'running' is a good idea. I've relied on version number
so folks can easily select the version of hadoop they want on the
cluster. Perhaps the best solution would be to allow an optional
parameter to the terminate script to specify the AMI ID if you need
extra certainty (the script already prompts with a list of instances
to terminate).
Tom
Re: Accessing S3 with Hadoop?
Posted by Tom White <to...@gmail.com>.
> I just updated the page to add a Notes section explaining the issue
> and referencing the JIRA issue # you mentioned earlier.
Great - thanks.
> > Are you able to do 'bin/hadoop-ec2 launch-cluster' then (on your workstation)
> >
> > . bin/hadoop-ec2-env.sh
> > ssh $SSH_OPTS "root@$MASTER_HOST" "sed -i -e
> > \"s/$MASTER_HOST/\$(hostname)/g\"
> > /usr/local/hadoop-$HADOOP_VERSION/conf/hadoop-site.xml"
> >
> > and then check to see if the master host has been set correctly (to
> > the internal IP) in the master host's hadoop-site.xml.
>
> Well, no, since my $MASTER_HOST is now just the external DNS name of
> the first instance started in the reservation, but this is performed
> as part of my launch-hadoop-cluster script. In any case, that value is
> not set to the internal IP, but rather to the hostname portion of the
> internal DNS name.
This is a bit of a mystery to me - I'll try to reproduce it in on my
workstation.
>
> Currently, my MR jobs are failing because the reducers can't copy the
> map output and I'm thinking it might be because there is some kind of
> external address getting in there somehow. I see connections to
> external IPs in netstat -tan (72.* addresses). Any ideas about that?
> In the hadoop-site.xml's on the slaves, the address is the external
> DNS name of the master (ec2-*) but that resolves to the internal 10/8
> address like it should.
>
> > Also, what version of the EC2 tools are you using?
>
> black:~/code/hadoop-0.14.0/src/contrib/ec2> ec2-version
> 1.2-11797 2007-03-01
> black:~/code/hadoop-0.14.0/src/contrib/ec2>
I'm using the same version so that's not it.
> > Instances are terminated on the basis of their AMI ID since 0.14.0.
> > See https://issues.apache.org/jira/browse/HADOOP-1504.
>
> I felt this was unsafe as it was, since it looked for a name of an
> image and then reversed it to the AMI ID. I just hacked it so you have
> to put in the AMI ID in hadoop-ec2-env.sh. Also, the script as it is
> right now doesn't grep for 'running' so may potentially shut down some
> instances starting up in another cluster. I may just be paranoid,
> however ;)
Checking for 'running' is a good idea. I've relied on version number
so folks can easily select the version of hadoop they want on the
cluster. Perhaps the best solution would be to allow an optional
parameter to the terminate script to specify the AMI ID if you need
extra certainty (the script already prompts with a list of instances
to terminate).
Tom
Re: Accessing S3 with Hadoop?
Posted by Ahad Rana <ah...@gmail.com>.
Hi Toby / Tom,
The failure to copy during the reduce phase is one of the side affects of
the bug specified in HADOOP-1783. You should apply Tom's patch and then see
if the failure condition still exists.
Thanks,
Ahad.
On 9/6/07, Toby DiPasquale <co...@gmail.com> wrote:
>
> On 9/6/07, Tom White <to...@gmail.com> wrote:
> > > Yeah, I actually read all of the wiki and your article about using
> > > Hadoop on EC2/S3 and I can't really find a reference to the S3 support
> > > not being for "regular" S3 keys. Did I miss something or should I
> > > update the wiki to make it more clear (or both)?
> >
> > I don't think this is explained clearly enough, so please do update
> > the wiki. Thanks.
>
> I just updated the page to add a Notes section explaining the issue
> and referencing the JIRA issue # you mentioned earlier.
>
> > > Also, the instructions on the EC2 page on the wiki no longer work, in
> > > that due to the kind of NAT Amazon is using, the slaves can't connect
> > > to the master using an externally-resolved IP address via a DNS name.
> > > What I mean is, if you set DNS to the external IP of your master
> > > instance, your slaves can resolve that address but cannot then connect
> > > to it. So, I had to alter the launch-hadoop-cluster and start-hadoop
> > > scripts and merge them to just pick the master and use its EC2-given
> > > name as the $MASTER_HOST to make it work.
> >
> > This sounds like the problem fixed in
> > https://issues.apache.org/jira/browse/HADOOP-1638 in 0.14.0, which is
> > the version you're using isn't it?
> >
> > Are you able to do 'bin/hadoop-ec2 launch-cluster' then (on your
> workstation)
> >
> > . bin/hadoop-ec2-env.sh
> > ssh $SSH_OPTS "root@$MASTER_HOST" "sed -i -e
> > \"s/$MASTER_HOST/\$(hostname)/g\"
> > /usr/local/hadoop-$HADOOP_VERSION/conf/hadoop-site.xml"
> >
> > and then check to see if the master host has been set correctly (to
> > the internal IP) in the master host's hadoop-site.xml.
>
> Well, no, since my $MASTER_HOST is now just the external DNS name of
> the first instance started in the reservation, but this is performed
> as part of my launch-hadoop-cluster script. In any case, that value is
> not set to the internal IP, but rather to the hostname portion of the
> internal DNS name.
>
> Currently, my MR jobs are failing because the reducers can't copy the
> map output and I'm thinking it might be because there is some kind of
> external address getting in there somehow. I see connections to
> external IPs in netstat -tan (72.* addresses). Any ideas about that?
> In the hadoop-site.xml's on the slaves, the address is the external
> DNS name of the master (ec2-*) but that resolves to the internal 10/8
> address like it should.
>
> > Also, what version of the EC2 tools are you using?
>
> black:~/code/hadoop-0.14.0/src/contrib/ec2> ec2-version
> 1.2-11797 2007-03-01
> black:~/code/hadoop-0.14.0/src/contrib/ec2>
>
> > > I also updated the scripts
> > > to only look for a given AMI ID and only start/manage/terminate
> > > instances of that AMI ID (since I have others I'd rather not
> > > terminated just on the basis of their AMI launch index ;-)).
> >
> > Instances are terminated on the basis of their AMI ID since 0.14.0.
> > See https://issues.apache.org/jira/browse/HADOOP-1504.
>
> I felt this was unsafe as it was, since it looked for a name of an
> image and then reversed it to the AMI ID. I just hacked it so you have
> to put in the AMI ID in hadoop-ec2-env.sh. Also, the script as it is
> right now doesn't grep for 'running' so may potentially shut down some
> instances starting up in another cluster. I may just be paranoid,
> however ;)
>
> --
> Toby DiPasquale
>
Re: Accessing S3 with Hadoop?
Posted by Toby DiPasquale <co...@gmail.com>.
On 9/6/07, Tom White <to...@gmail.com> wrote:
> > Yeah, I actually read all of the wiki and your article about using
> > Hadoop on EC2/S3 and I can't really find a reference to the S3 support
> > not being for "regular" S3 keys. Did I miss something or should I
> > update the wiki to make it more clear (or both)?
>
> I don't think this is explained clearly enough, so please do update
> the wiki. Thanks.
I just updated the page to add a Notes section explaining the issue
and referencing the JIRA issue # you mentioned earlier.
> > Also, the instructions on the EC2 page on the wiki no longer work, in
> > that due to the kind of NAT Amazon is using, the slaves can't connect
> > to the master using an externally-resolved IP address via a DNS name.
> > What I mean is, if you set DNS to the external IP of your master
> > instance, your slaves can resolve that address but cannot then connect
> > to it. So, I had to alter the launch-hadoop-cluster and start-hadoop
> > scripts and merge them to just pick the master and use its EC2-given
> > name as the $MASTER_HOST to make it work.
>
> This sounds like the problem fixed in
> https://issues.apache.org/jira/browse/HADOOP-1638 in 0.14.0, which is
> the version you're using isn't it?
>
> Are you able to do 'bin/hadoop-ec2 launch-cluster' then (on your workstation)
>
> . bin/hadoop-ec2-env.sh
> ssh $SSH_OPTS "root@$MASTER_HOST" "sed -i -e
> \"s/$MASTER_HOST/\$(hostname)/g\"
> /usr/local/hadoop-$HADOOP_VERSION/conf/hadoop-site.xml"
>
> and then check to see if the master host has been set correctly (to
> the internal IP) in the master host's hadoop-site.xml.
Well, no, since my $MASTER_HOST is now just the external DNS name of
the first instance started in the reservation, but this is performed
as part of my launch-hadoop-cluster script. In any case, that value is
not set to the internal IP, but rather to the hostname portion of the
internal DNS name.
Currently, my MR jobs are failing because the reducers can't copy the
map output and I'm thinking it might be because there is some kind of
external address getting in there somehow. I see connections to
external IPs in netstat -tan (72.* addresses). Any ideas about that?
In the hadoop-site.xml's on the slaves, the address is the external
DNS name of the master (ec2-*) but that resolves to the internal 10/8
address like it should.
> Also, what version of the EC2 tools are you using?
black:~/code/hadoop-0.14.0/src/contrib/ec2> ec2-version
1.2-11797 2007-03-01
black:~/code/hadoop-0.14.0/src/contrib/ec2>
> > I also updated the scripts
> > to only look for a given AMI ID and only start/manage/terminate
> > instances of that AMI ID (since I have others I'd rather not
> > terminated just on the basis of their AMI launch index ;-)).
>
> Instances are terminated on the basis of their AMI ID since 0.14.0.
> See https://issues.apache.org/jira/browse/HADOOP-1504.
I felt this was unsafe as it was, since it looked for a name of an
image and then reversed it to the AMI ID. I just hacked it so you have
to put in the AMI ID in hadoop-ec2-env.sh. Also, the script as it is
right now doesn't grep for 'running' so may potentially shut down some
instances starting up in another cluster. I may just be paranoid,
however ;)
--
Toby DiPasquale
Re: Accessing S3 with Hadoop?
Posted by Tom White <to...@gmail.com>.
> Yeah, I actually read all of the wiki and your article about using
> Hadoop on EC2/S3 and I can't really find a reference to the S3 support
> not being for "regular" S3 keys. Did I miss something or should I
> update the wiki to make it more clear (or both)?
I don't think this is explained clearly enough, so please do update
the wiki. Thanks.
> Also, the instructions on the EC2 page on the wiki no longer work, in
> that due to the kind of NAT Amazon is using, the slaves can't connect
> to the master using an externally-resolved IP address via a DNS name.
> What I mean is, if you set DNS to the external IP of your master
> instance, your slaves can resolve that address but cannot then connect
> to it. So, I had to alter the launch-hadoop-cluster and start-hadoop
> scripts and merge them to just pick the master and use its EC2-given
> name as the $MASTER_HOST to make it work.
This sounds like the problem fixed in
https://issues.apache.org/jira/browse/HADOOP-1638 in 0.14.0, which is
the version you're using isn't it?
Are you able to do 'bin/hadoop-ec2 launch-cluster' then (on your workstation)
. bin/hadoop-ec2-env.sh
ssh $SSH_OPTS "root@$MASTER_HOST" "sed -i -e
\"s/$MASTER_HOST/\$(hostname)/g\"
/usr/local/hadoop-$HADOOP_VERSION/conf/hadoop-site.xml"
and then check to see if the master host has been set correctly (to
the internal IP) in the master host's hadoop-site.xml.
Also, what version of the EC2 tools are you using?
> I also updated the scripts
> to only look for a given AMI ID and only start/manage/terminate
> instances of that AMI ID (since I have others I'd rather not
> terminated just on the basis of their AMI launch index ;-)).
Instances are terminated on the basis of their AMI ID since 0.14.0.
See https://issues.apache.org/jira/browse/HADOOP-1504.
Tom
Re: Accessing S3 with Hadoop?
Posted by Toby DiPasquale <co...@gmail.com>.
On 9/5/07, Tom White <to...@gmail.com> wrote:
> There is an open Jira issue for adding support for reading regular S3
> files (https://issues.apache.org/jira/browse/HADOOP-930) which would
> solve this problem. Until this is implemented Ahad's suggestion is a
> good workaround (you can use distcp too).
Yeah, I actually read all of the wiki and your article about using
Hadoop on EC2/S3 and I can't really find a reference to the S3 support
not being for "regular" S3 keys. Did I miss something or should I
update the wiki to make it more clear (or both)?
Also, the instructions on the EC2 page on the wiki no longer work, in
that due to the kind of NAT Amazon is using, the slaves can't connect
to the master using an externally-resolved IP address via a DNS name.
What I mean is, if you set DNS to the external IP of your master
instance, your slaves can resolve that address but cannot then connect
to it. So, I had to alter the launch-hadoop-cluster and start-hadoop
scripts and merge them to just pick the master and use its EC2-given
name as the $MASTER_HOST to make it work. I also updated the scripts
to only look for a given AMI ID and only start/manage/terminate
instances of that AMI ID (since I have others I'd rather not
terminated just on the basis of their AMI launch index ;-)).
--
Toby DiPasquale
Re: Accessing S3 with Hadoop?
Posted by Tom White <to...@gmail.com>.
There is an open Jira issue for adding support for reading regular S3
files (https://issues.apache.org/jira/browse/HADOOP-930) which would
solve this problem. Until this is implemented Ahad's suggestion is a
good workaround (you can use distcp too).
Tom
On 05/09/07, Ahad Rana <ah...@gmail.com> wrote:
> Hi Toby,
>
> Keep in mind that in the S3 Filesystem is a block based file system
> implementation that is built on top of the S3 Service. Thus, if you store a
> file named foo using the hadoop tools in an S3 Bucket, the underlying code
> will store metadata (block information + hadoop specific headers) in the key
> /user/root/foo. Actual data will be stored under under one or more keys of
> the form: /user/root/BLOCK_[BLOCKID], with each block representing at the
> maximum a BLOCK SIZE portion (32 MB by default) of the file's content.
>
> So, this means that you cannot just copy files into an S3 bucket (from
> another tool) and have them be visible via the S3 FileSystem support in
> hadoop. You would have to use something like bin/hadoop fs -fs s3://bucket
> -copyFromLocal localfile /user/root/s3file to copy files across into S3 in
> the hadoop specific format.
>
> Hope that helps.
>
> Ahad.
>
> On 9/5/07, Toby DiPasquale <co...@gmail.com> wrote:
> >
> > Hi all,
> >
> > I'm trying to get Hadoop 0.14.0 to talk to S3 but its not working out
> > for me. My credentials are definitely right (checked multiple times
> > and that's not the issue in any case) and the bucket I'm querying
> > definitely does exist already and has keys under it.
> >
> > However, when I try to access S3, I get really weird stuff going on.
> > If I try this:
> >
> > bin/hadoop fs -ls key-with-stuff-under-it
> >
> > it fails. I set it to DEBUG and I can see that the S3 library is
> > trying to query:
> >
> > GET /BUCKET/%2Fuser%2Froot%2Fkey-with-stuff-under-it
> >
> > which is wrong for obvious reasons, but which I can sort of
> > understand, as the default value of the workingDir instance variable
> > in S3FileSystem.java is "/user/${user.name}" (anybody know where this
> > default comes from, btw? it make no sense to me as is).
> >
> > When I try the full URI:
> >
> > bin/hadoop fs -ls s3://BUCKET/key-with-stuff-under-it
> >
> > it still fails, instead doing this even stranger thing and trying to query
> >
> > GET /BUCKET/%2Fkey-with-stuff-under-it
> >
> > Note: the embedded %2F in there that makes it incorrect.
> >
> > It also does this same thing whether I embed the credentials in the
> > full s3:// URI or not.
> >
> > Then, I thought maybe it doesn't want a bucket at all in
> > fs.default.name, so I took that out and then the configuration failed
> > to parse on startup, throwing an IllegalArgumentException, telling me
> > it was expecting an authority portion to the URI it was given.
> >
> > Are the docs on the wiki and Amazon's articles wrong about how to
> > configure hadoop-site.xml for S3 as a filesystem? Or am I doing
> > something wrong in my config, leaving something out, etc? Any
> > thoughts? Should I be setting the workingDir somewhere in the config
> > to something more sensible? I couldn't find much about this on the
> > mailing list or the forums at Amazon so I thought I'd just ask. My
> > config info is below.
> >
> > Any help is greatly appreciated. Thanks :)
> >
> > My mapred-default.xml is the default one from
> > hadoop-0.14.0/src/contrib/ec2/image/hadoop-init. I am running this on
> > EC2 with just MapReduce (no HDFS NameNodes/DataNodes running). Here's
> > my hadoop-site.xml (anonymized, of course):
> >
> > ------------------------------------
> > <?xml version="1.0"?>
> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >
> > <configuration>
> >
> > <property>
> > <name>hadoop.tmp.dir</name>
> > <value>/mnt/hadoop</value>
> > </property>
> >
> > <property>
> > <name>fs.default.name</name>
> > <value>s3://BUCKET</value>
> > </property>
> >
> > <property>
> > <name>fs.s3.awsAccessKeyId</name>
> > <value>XXXXXXXXXXXXXXXXXX</value>
> > </property>
> >
> > <property>
> > <name>fs.s3.awsSecretAccessKey</name>
> > <value>XXXXXXXXXXXXXXXXXX</value>
> > </property>
> >
> > <property>
> > <name>mapred.job.tracker</name>
> > <value>$MASTER_HOST:50002</value>
> > </property>
> >
> > </configuration>
> > ------------------------------------
> >
> > --
> > Toby DiPasquale
> >
>
Re: Accessing S3 with Hadoop?
Posted by Ahad Rana <ah...@gmail.com>.
Hi Toby,
Keep in mind that in the S3 Filesystem is a block based file system
implementation that is built on top of the S3 Service. Thus, if you store a
file named foo using the hadoop tools in an S3 Bucket, the underlying code
will store metadata (block information + hadoop specific headers) in the key
/user/root/foo. Actual data will be stored under under one or more keys of
the form: /user/root/BLOCK_[BLOCKID], with each block representing at the
maximum a BLOCK SIZE portion (32 MB by default) of the file's content.
So, this means that you cannot just copy files into an S3 bucket (from
another tool) and have them be visible via the S3 FileSystem support in
hadoop. You would have to use something like bin/hadoop fs -fs s3://bucket
-copyFromLocal localfile /user/root/s3file to copy files across into S3 in
the hadoop specific format.
Hope that helps.
Ahad.
On 9/5/07, Toby DiPasquale <co...@gmail.com> wrote:
>
> Hi all,
>
> I'm trying to get Hadoop 0.14.0 to talk to S3 but its not working out
> for me. My credentials are definitely right (checked multiple times
> and that's not the issue in any case) and the bucket I'm querying
> definitely does exist already and has keys under it.
>
> However, when I try to access S3, I get really weird stuff going on.
> If I try this:
>
> bin/hadoop fs -ls key-with-stuff-under-it
>
> it fails. I set it to DEBUG and I can see that the S3 library is
> trying to query:
>
> GET /BUCKET/%2Fuser%2Froot%2Fkey-with-stuff-under-it
>
> which is wrong for obvious reasons, but which I can sort of
> understand, as the default value of the workingDir instance variable
> in S3FileSystem.java is "/user/${user.name}" (anybody know where this
> default comes from, btw? it make no sense to me as is).
>
> When I try the full URI:
>
> bin/hadoop fs -ls s3://BUCKET/key-with-stuff-under-it
>
> it still fails, instead doing this even stranger thing and trying to query
>
> GET /BUCKET/%2Fkey-with-stuff-under-it
>
> Note: the embedded %2F in there that makes it incorrect.
>
> It also does this same thing whether I embed the credentials in the
> full s3:// URI or not.
>
> Then, I thought maybe it doesn't want a bucket at all in
> fs.default.name, so I took that out and then the configuration failed
> to parse on startup, throwing an IllegalArgumentException, telling me
> it was expecting an authority portion to the URI it was given.
>
> Are the docs on the wiki and Amazon's articles wrong about how to
> configure hadoop-site.xml for S3 as a filesystem? Or am I doing
> something wrong in my config, leaving something out, etc? Any
> thoughts? Should I be setting the workingDir somewhere in the config
> to something more sensible? I couldn't find much about this on the
> mailing list or the forums at Amazon so I thought I'd just ask. My
> config info is below.
>
> Any help is greatly appreciated. Thanks :)
>
> My mapred-default.xml is the default one from
> hadoop-0.14.0/src/contrib/ec2/image/hadoop-init. I am running this on
> EC2 with just MapReduce (no HDFS NameNodes/DataNodes running). Here's
> my hadoop-site.xml (anonymized, of course):
>
> ------------------------------------
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <configuration>
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/mnt/hadoop</value>
> </property>
>
> <property>
> <name>fs.default.name</name>
> <value>s3://BUCKET</value>
> </property>
>
> <property>
> <name>fs.s3.awsAccessKeyId</name>
> <value>XXXXXXXXXXXXXXXXXX</value>
> </property>
>
> <property>
> <name>fs.s3.awsSecretAccessKey</name>
> <value>XXXXXXXXXXXXXXXXXX</value>
> </property>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>$MASTER_HOST:50002</value>
> </property>
>
> </configuration>
> ------------------------------------
>
> --
> Toby DiPasquale
>