You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/05/08 16:55:16 UTC
[GitHub] [flink] sjwiesman commented on a change in pull request #12043: [hotfix][docs][s3] Clarify wording of S3 Filesystem support
sjwiesman commented on a change in pull request #12043:
URL: https://github.com/apache/flink/pull/12043#discussion_r422252657
##########
File path: docs/ops/filesystems/s3.md
##########
@@ -60,10 +60,15 @@ Flink provides two file systems to talk to Amazon S3, `flink-s3-fs-presto` and `
Both implementations are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
- `flink-s3-fs-presto`, registered under the scheme *s3://* and *s3p://*, is based on code from the [Presto project](https://prestodb.io/).
- You can configure it the same way you can [configure the Presto file system](https://prestodb.io/docs/0.187/connector/hive.html#amazon-s3-configuration) by placing adding the configurations to your `flink-conf.yaml`. Presto is the recommended file system for checkpointing to S3.
+ You can configure it using [the same configuration keys as the Preso file system](https://prestodb.io/docs/0.187/connector/hive.html#amazon-s3-configuration), by adding the configurations to your `flink-conf.yaml`. The Presto S3 implementation is the recommended file system for checkpointing to S3.
Review comment:
```suggestion
You can configure it using [the same configuration keys as the Presto file system](https://prestodb.io/docs/0.187/connector/hive.html#amazon-s3-configuration), by adding the configurations to your `flink-conf.yaml`. The Presto S3 implementation is the recommended file system for checkpointing to S3.
```
##########
File path: docs/ops/filesystems/s3.md
##########
@@ -60,10 +60,15 @@ Flink provides two file systems to talk to Amazon S3, `flink-s3-fs-presto` and `
Both implementations are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
- `flink-s3-fs-presto`, registered under the scheme *s3://* and *s3p://*, is based on code from the [Presto project](https://prestodb.io/).
- You can configure it the same way you can [configure the Presto file system](https://prestodb.io/docs/0.187/connector/hive.html#amazon-s3-configuration) by placing adding the configurations to your `flink-conf.yaml`. Presto is the recommended file system for checkpointing to S3.
+ You can configure it using [the same configuration keys as the Preso file system](https://prestodb.io/docs/0.187/connector/hive.html#amazon-s3-configuration), by adding the configurations to your `flink-conf.yaml`. The Presto S3 implementation is the recommended file system for checkpointing to S3.
- `flink-s3-fs-hadoop`, registered under *s3://* and *s3a://*, based on code from the [Hadoop Project](https://hadoop.apache.org/).
- The file system can be [configured exactly like Hadoop's s3a](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A) by placing adding the configurations to your `flink-conf.yaml`. It is the only S3 file system with support for the [StreamingFileSink]({{ site.baseurl}}/dev/connectors/streamfile_sink.html).
+ The file system can be [configured using Hadoop's s3a configuration keys](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A) by adding the configurations to your `flink-conf.yaml`.
+
+ For example, Hadoop has a `fs.s3a.connection.maximum` configuration key. If you want to change it, you need to put `s3.connection.maximum: xyz` to the `flink-conf.yaml` (Flink will internally translate this back to `fs.s3a.connection.maximum`. There is no need to pass configuration parameters using Hadoop's XML configuration files).
Review comment:
I don't think this last part needs to be in parenthesis
```suggestion
For example, Hadoop has a `fs.s3a.connection.maximum` configuration key. If you want to change it, you need to put `s3.connection.maximum: xyz` to the `flink-conf.yaml`. Flink will internally translate this back to `fs.s3a.connection.maximum`. There is no need to pass configuration parameters using Hadoop's XML configuration files.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org