Hadoop 0.23.3 and Amazon S3

[Alexander Hristov <al...@planetalia.com>]

Hi Again

I have problems trying to make Hadoop use S3 or S3N as filesystem.

This is what I have in core-site.xml:

<configuration>
     <property>
       <name>fs.default.name</name>
       <value>s3n://bucketname</value>
     </property>

     <property>
       <name>fs.s3.awsAccessKeyId</name>
       <value> something <value>
     </property>

     <property>
       <name>fs.s3.awsSecretAccessKey</name>
       <value> something </value>
     </property>

     <property>
       <name>fs.s3n.awsAccessKeyId</name>
       <value> something </value>
     </property>

     <property>
       <name>fs.s3n.awsSecretAccessKey</name>
       <value> something </value>
     </property>

      <property>
         <name>hadoop.tmp.dir</name>
         <value>/tmp/hadoop</value>
      </property>
</configuration>

The secret key does not contain any slashes.

When I use s3n://buckename, I get this:

[hadoop@ahristov hadoop]$ hadoop fs -put LICENSE.txt /
put: org.jets3t.service.S3ServiceException: S3 HEAD request failed for 
'/LICENSE.txt' - ResponseCode=403, ResponseMessage=Forbidden

And when I use s3://bucketname, I get this:

[hadoop@ahristov hadoop]$ hadoop fs -put LICENSE.txt /
put: `/': No such file or directory

I couldn't find any logs generated anywhere.

On the other hand, if I use a quick and dirty Java snippet to achieve 
the same, like:

         Configuration conf = new Configuration();
conf.addResource(TestS3.class.getResourceAsStream("/res/core-s3.xml"));
         FileSystem fileSystem = FileSystem.get(conf);
         InputStream in = TestS3.class.getResourceAsStream("/res/test.txt");
         FSDataOutputStream out = fileSystem.create(new Path("/book.txt"));
         byte[] buffer = new byte[10240];
         while (true) {
             int read= in.read(buffer);
             if (read== -1) break;
             out.write(buffer,0,read);
         }
         out.close();
         in.close();

It works both with s3:// and s3n://

Regards

Alexander