You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by ra...@apache.org on 2017/02/17 14:50:08 UTC
[28/50] [abbrv] hadoop git commit: HADOOP-14019. Fix some typos in
the s3a docs. Contributed by Steve Loughran
HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/bdad8b7b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/bdad8b7b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/bdad8b7b
Branch: refs/heads/HDFS-10285
Commit: bdad8b7b97d7f48119f016d68f32982d680c8796
Parents: f432999
Author: Mingliang Liu <li...@apache.org>
Authored: Thu Feb 16 16:41:31 2017 -0800
Committer: Mingliang Liu <li...@apache.org>
Committed: Thu Feb 16 16:41:31 2017 -0800
----------------------------------------------------------------------
.../src/site/markdown/tools/hadoop-aws/index.md | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/bdad8b7b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
index 2471a52..0ff314c 100644
--- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
+++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
@@ -970,7 +970,7 @@ This is because the property values are kept in these files, and cannot be
dynamically patched.
Instead, callers need to create different configuration files for each
-bucket, setting the base secrets (`fs.s3a.bucket.nightly.access.key`, etc),
+bucket, setting the base secrets (`fs.s3a.access.key`, etc),
then declare the path to the appropriate credential file in
a bucket-specific version of the property `fs.s3a.security.credential.provider.path`.
@@ -1044,7 +1044,7 @@ declaration. For example:
### <a name="s3a_fast_upload"></a>Stabilizing: S3A Fast Upload
-**New in Hadoop 2.7; significantly enhanced in Hadoop 2.9**
+**New in Hadoop 2.7; significantly enhanced in Hadoop 2.8**
Because of the nature of the S3 object store, data written to an S3A `OutputStream`
@@ -1204,8 +1204,18 @@ consumed, and so eliminates heap size as the limiting factor in queued uploads
<value>disk</value>
</property>
+<property>
+ <name>fs.s3a.buffer.dir</name>
+ <value></value>
+ <description>Comma separated list of temporary directories use for
+ storing blocks of data prior to their being uploaded to S3.
+ When unset, the Hadoop temporary directory hadoop.tmp.dir is used</description>
+</property>
+
```
+This is the default buffer mechanism. The amount of data which can
+be buffered is limited by the amount of available disk space.
#### <a name="s3a_fast_upload_bytebuffer"></a>Fast Upload with ByteBuffers: `fs.s3a.fast.upload.buffer=bytebuffer`
@@ -1219,7 +1229,7 @@ The amount of data which can be buffered is
limited by the Java runtime, the operating system, and, for YARN applications,
the amount of memory requested for each container.
-The slower the write bandwidth to S3, the greater the risk of running out
+The slower the upload bandwidth to S3, the greater the risk of running out
of memory \u2014and so the more care is needed in
[tuning the upload settings](#s3a_fast_upload_thread_tuning).
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org