[Hadoop Wiki] Update of "AmazonS3" by SteveLoughran

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "AmazonS3" page has been changed by SteveLoughran:
https://wiki.apache.org/hadoop/AmazonS3?action=diff&rev1=14&rev2=15

Comment:
update this doc to the hadoop 1.x+ codebase and spell out that you can't swap out HDFS for S3

  Hadoop provides two filesystems that use S3.
  
   S3 Native FileSystem (URI scheme: s3n)::
-  A native filesystem for reading and writing regular files on S3. The advantage of this filesystem is that you can access files on S3 that were written with other tools. Conversely, other tools can access files written using Hadoop. The disadvantage is the 5GB limit on file size imposed by S3. For this reason it is not suitable as a replacement for HDFS (which has support for very large files).
+  A native filesystem for reading and writing regular files on S3. The advantage of this filesystem is that you can access files on S3 that were written with other tools. Conversely, other tools can access files written using Hadoop. The disadvantage is the 5GB limit on file size imposed by S3.
  
   S3 Block FileSystem (URI scheme: s3)::
   A block-based filesystem backed by S3. Files are stored as blocks, just like they are in HDFS. This permits efficient implementation of renames. This filesystem requires you to dedicate a bucket for the filesystem - you should not use an existing bucket containing files, or write other files to the same bucket. The files stored by this filesystem can be larger than 5GB, but they are not interoperable with other S3 tools.
  
- There are two ways that S3 can be used with Hadoop's Map/Reduce, either as a replacement for HDFS using the S3 block filesystem
- (i.e. using it as a reliable distributed filesystem with support for very large files)
- or as a convenient repository for data input to and output from MapReduce, using either S3 filesystem. In the second case
- HDFS is still used for the Map/Reduce phase. Note also, that by using S3 as an input to MapReduce you lose the data locality optimization, which may be significant.
+ S3 can be used as a convenient repository for data input to and output for analytics applications using either S3 filesystem.
+ Data in S3 outlasts Hadoop clusters on EC2, so they should be where persistent data must be kept.
+ 
+ Note that by using S3 as an input you lose the data locality optimization, which may be significant. The general best practise is to copy in data using `distcp` at the start of a workflow, then copy it out at the end, using the transient HDFS in between.
  
  = History =
   * The S3 block filesystem was introduced in Hadoop 0.10.0 ([[http://issues.apache.org/jira/browse/HADOOP-574|HADOOP-574]]), but this had a few bugs so you should use Hadoop 0.10.1 or later.
   * The S3 native filesystem was introduced in Hadoop 0.18.0 ([[http://issues.apache.org/jira/browse/HADOOP-930|HADOOP-930]]) and rename support was added in Hadoop 0.19.0 ([[https://issues.apache.org/jira/browse/HADOOP-3361|HADOOP-3361]]).
  
- = Setting up hadoop to use S3 as a replacement for HDFS =
+ = Why you cannot use S3 as a replacement for HDFS =
  
- Put the following in ''conf/hadoop-site.xml'' to set the default filesystem to be the S3 block filesystem:
+ You cannot use either of the S3 filesystems as a drop-in replacement for HDFS. Amazon S3 is an "object store" with
+  * eventual consistency: changes made by one application (creation, updates and deletions) will not be visible until some undefined time.
+  * s3n: non-atomic rename and delete operations. Renaming or deleting large directories takes time proportional to the number of entries -and visible to other processes during this time, and indeed, until the eventual consistency has been resolved.
+ 
+ S3 is not a filesystem. The Hadoop S3 filesystem bindings make it pretend to be a filesystem, but it is not. It can
+ act as a source of data, and as a destination -though in the latter case, you must remember that the output may not be immediately visible.
+ 
+ == Configuring to use s3/ s3n filesystems ==
+ 
+ Edit your `core-site.xml` file to include your S3 keys
  
  {{{
- <property>
-   <name>fs.default.name</name>
-   <value>s3://BUCKET</value>
- </property>
  
  <property>
    <name>fs.s3.awsAccessKeyId</name>
@@ -42, +47 @@

  </property>
  }}}
  
+ You can then use URLs to your bucket : ``s3n://MYBUCKET/``, or directories and files inside it.
- For the S3 native filesystem, just replace `s3` with `s3n` in the above.
- 
- Note that you can not use s3n as a replacement for HDFS on Hadoop versions prior to 0.19.0 since rename was not supported.
- 
- Alternatively, you can put the access key ID and the secret access key into a ''s3'' (or ''s3n'') URI as the user info:
  
  {{{
- <property>
-   <name>fs.default.name</name>
-   <value>s3://ID:SECRET@BUCKET</value>
- </property>
+ 
+ s3n://BUCKET/
+ s3n://BUCKET/dir
+ s3n://BUCKET/dir/files.csv.tar.gz
+ s3n://BUCKET/dir/*.gz
+ 
+ }}}
+ 
+ Alternatively, you can put the access key ID and the secret access key into a ''s3n'' (or ''s3'') URI as the user info:
+ 
+ {{{
+   s3n://ID:SECRET@BUCKET
  }}}
  
  Note that since the secret
@@ -60, +69 @@

  Keys specified in the URI take precedence over any specified using the properties `fs.s3.awsAccessKeyId` and
  `fs.s3.awsSecretAccessKey`.
  
+ This option is less secure as the URLs are likely to appear in output logs and error messages, so being exposed to remote users.
- Running the Map/Reduce demo in the [[http://hadoop.apache.org/core/api/index.html|Hadoop API Documentation]] using
- S3 is now a matter of running:
- 
- {{{
- mkdir input
- cp conf/*.xml input
- bin/hadoop fs -put input input
- bin/hadoop fs -ls input
- bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
- bin/hadoop fs -get output output
- cat output/*
- }}}
- 
- To run in distributed mode you only need to run the MapReduce daemons (JobTracker and [[TaskTracker]]s) - HDFS NameNode and [[DataNode]]s are unnecessary.
- 
- {{{
- bin/start-mapred.sh
- }}}
  
  = Security =
  
@@ -100, +92 @@

  
  Other schemes supported by `distcp` are `file` (for local), and `http`.
  
- You'll likely encounter the following errors if you are running a stock Hadoop 0.11.X.
- 
- {{{
- org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed...We encountered an internal error. Please try again...
- 
- put: Input stream is not repeatable as 1048576 bytes have been written, exceeding the available buffer size of 131072
- }}}
- 
- See [[https://issues.apache.org/jira/browse/HADOOP-882|HADOOP-882]] for discussion of the above issues and workarounds/fixes.
- 
- = S3 Block FileSystem Version Numbers =
- From release 0.13.0 the S3 block filesystem stores a version number in the file metadata. This table lists the first Hadoop release for each version number.
- ||'''Version number'''||'''Release'''||
- ||Unversioned         ||0.10.0       ||
- ||1                   ||0.13.0       ||
- 
- There is a migration tool (`org.apache.hadoop.fs.s3.MigrationTool`) that can be used to migrate data to the latest version. Run it without arguments to get usage instructions.
-