You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Samrat002 (via GitHub)" <gi...@apache.org> on 2026/04/16 02:46:00 UTC
[PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Samrat002 opened a new pull request, #27937:
URL: https://github.com/apache/flink/pull/27937
## What is the purpose of the change
Add documentation for Native s3 FileSystem
## Brief change log
Add documentation and show how to use new s3Filesystem.
## Verifying this change
Build the docs in local using Hugo
```
~/C/O/flink/docs │ on FLINK-39118-…ocumentation !1 ▓▒░ docker run -v $(pwd):/src -p 1313:1313 jakejarvis/hugo-extended:latest server --buildDrafts --buildFuture --bind 0.0.0.0
Watching for changes in /src/{assets,content,content.zh,data,layouts,static,themes}
Watching for config changes in /src/config.toml, /src/themes/connectors/config.yaml
Start building sites …
hugo v0.124.1-db083b05f16c945fec04f745f0ca8640560cf1ec+extended linux/arm64 BuildDate=2024-03-20T11:40:10Z VendorInfo=docker
| EN | ZH
-------------------+-----+------
Pages | 502 | 500
Paginator pages | 0 | 0
Non-page files | 0 | 0
Static files | 266 | 266
Processed images | 0 | 0
Aliases | 419 | 416
Cleaned | 0 | 0
Built in 7938 ms
Environment: "development"
Serving pages from disk
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at //localhost:1313/flink/flink-docs-master/ (bind address 0.0.0.0)
Press Ctrl+C to stop
```
<img width="1504" height="827" alt="Screenshot 2026-04-15 at 10 37 11 PM" src="https://github.com/user-attachments/assets/688d2b62-c8ed-4cac-9052-b230c6a63bd9" />
<img width="1498" height="842" alt="Screenshot 2026-04-15 at 10 37 24 PM" src="https://github.com/user-attachments/assets/c2e93b23-95b8-4066-9b55-eae56292818c" />
#### Chinese Doc Render :
<img width="1293" height="907" alt="Screenshot 2026-04-15 at 10 37 48 PM" src="https://github.com/user-attachments/assets/e0ccba3a-f248-4ed1-9afc-593d3e849e96" />
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)
- The serializers: (yes / no / don't know)
- The runtime per-record code paths (performance sensitive): (yes / no / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
- The S3 file system connector: (yes / no / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / no) no
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3098703443
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,208 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations:
-{{< hint info >}}
-You don't have to configure this manually if you are running [Flink on EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html).
-{{< /hint >}}
+| Implementation | Checkpointing | FileSink | Notes |
+|---------------|:---:|:---:|-------|
+| **Native S3** (`flink-s3-fs-native`) | ✓ | ✓ | **Experimental** in Flink 2.3. Built on AWS SDK v2; no Hadoop dependency. |
+| **Presto S3** (`flink-s3-fs-presto`) | ✓ | x | Production-proven for checkpointing. |
+| **Hadoop S3** (`flink-s3-fs-hadoop`) | ✓ | ✓ | Mature; the only stable implementation that provides `RecoverableWriter` for the FileSink. |
Review Comment:
maybe have a column for RecoverableWriter implementation. It would be good to be explicit around what the RecoverableWriter bring as opposed to checkpointing - a compare and contract of the capabilities to inform the user. Is there an intension this is a good thing that the native one will also implement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3093807080
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code, recommended for checkpointing.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code, has FileSystem sink support.
+
+All three are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+
+## Common Configuration
+
+### Configure Access Credentials
+
+After setting up the S3 FileSystem implementation, you need to make sure that Flink is allowed to access your S3 buckets.
+
+#### Identity and Access Management (IAM) (Recommended)
+
+The recommended way of setting up credentials on AWS is via [Identity and Access Management (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html). You can use IAM features to securely give Flink instances the credentials that they need to access S3 buckets. Details about how to do this are beyond the scope of this documentation. Please refer to the AWS user guide. What you are looking for are [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+
+If you set this up correctly, you can manage access to S3 within AWS and don't need to distribute any access keys to Flink.
+
+#### Delegation Tokens
+
+[Delegation tokens]({{< ref "docs/deployment/security/security-delegation-token" >}}) provide time-bounded, automatically negotiated credentials. The JobManager uses long-lived credentials to call AWS STS and obtain short-lived session tokens, which are then automatically distributed to TaskManagers.
Review Comment:
Thanks for the suggestion! I've added a parenthetical "(access key and secret key)" to the intro sentence, and the YAML block immediately below shows the exact config keys.
I think this level of detail is sufficient. Adding more would risk duplicating what's already in the next paragraph.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3091165289
##########
docs/content.zh/docs/deployment/filesystems/s3.md:
##########
@@ -50,127 +48,321 @@ env.fromSource(
"s3-input"
);
-// 写入 S3 bucket
+// Write to S3 bucket
stream.sinkTo(
- FileSink.forRowFormat(
- new Path("s3://<bucket>/<endpoint>"), new SimpleStringEncoder<>()
- ).build()
+ FileSink.forRowFormat(
+ new Path("s3://<bucket>/<endpoint>"), new SimpleStringEncoder<>()
+ ).build()
);
-
-// 使用 S3 作为 checkpoint storage
+// Use S3 as checkpoint storage
Configuration config = new Configuration();
config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "s3://<your-bucket>/<endpoint>");
env.configure(config);
```
-注意这些例子并*不详尽*，S3 同样可以用在其他场景，包括 [JobManager 高可用配置]({{< ref "docs/deployment/ha/overview" >}}) 或 [RocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend)，以及所有 Flink 需要使用文件系统 URI 的位置。
+Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
Review Comment:
As this is the Chinese counterpart for the doc, could you rather add a translated version. Probably AI could help.
Also there are some existing Chinese, being reverted to english. Could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3092278175
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
Review Comment:
nit: hadoop -> Hadoop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3098868117
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,208 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations:
-{{< hint info >}}
-You don't have to configure this manually if you are running [Flink on EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html).
-{{< /hint >}}
+| Implementation | Checkpointing | FileSink | Notes |
+|---------------|:---:|:---:|-------|
+| **Native S3** (`flink-s3-fs-native`) | ✓ | ✓ | **Experimental** in Flink 2.3. Built on AWS SDK v2; no Hadoop dependency. |
+| **Presto S3** (`flink-s3-fs-presto`) | ✓ | x | Production-proven for checkpointing. |
+| **Hadoop S3** (`flink-s3-fs-hadoop`) | ✓ | ✓ | Mature; the only stable implementation that provides `RecoverableWriter` for the FileSink. |
-Flink provides two file systems to talk to Amazon S3, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
-Both implementations are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+Previously, users had to choose between Presto (recommended for checkpointing throughput) and Hadoop (the only implementation with `RecoverableWriter`, required by the [FileSink]({{< ref "docs/connectors/datastream/filesystem" >}})). The Native S3 implementation unifies both capabilities in a single plugin and [benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to Presto at state sizes up to 15 GB.
Review Comment:
Maybe we can be less exact on numbers since it can change rapidly when we add new features or users have difference use-case. We can mention that measurements show performance gain and stability without exact details. This doc is for sales and how to use. If users are interested in the details (majority don't) they can find it manually without links.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3092366990
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code, recommended for checkpointing.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code, has FileSystem sink support.
+
+All three are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+
+## Common Configuration
+
+### Configure Access Credentials
+
+After setting up the S3 FileSystem implementation, you need to make sure that Flink is allowed to access your S3 buckets.
+
+#### Identity and Access Management (IAM) (Recommended)
+
+The recommended way of setting up credentials on AWS is via [Identity and Access Management (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html). You can use IAM features to securely give Flink instances the credentials that they need to access S3 buckets. Details about how to do this are beyond the scope of this documentation. Please refer to the AWS user guide. What you are looking for are [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+
+If you set this up correctly, you can manage access to S3 within AWS and don't need to distribute any access keys to Flink.
+
+#### Delegation Tokens
+
+[Delegation tokens]({{< ref "docs/deployment/security/security-delegation-token" >}}) provide time-bounded, automatically negotiated credentials. The JobManager uses long-lived credentials to call AWS STS and obtain short-lived session tokens, which are then automatically distributed to TaskManagers.
Review Comment:
could we add a hyper link for `AWS STS`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #27937:
URL: https://github.com/apache/flink/pull/27937#issuecomment-4273165781
Hi @RocMarshal, done! I've cherry-picked the commit from the master branch (565dbbccf8c). Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3092331154
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
Review Comment:
I am not sure what you mean by both checkpointing and the FileSystem sink. I assume it is the same as the next sentence, if so we should remove this sentence.
nit: as well as FileSystem sink -> as well as the FileSystem sink.
Some questions that came up for me:
- Are there any types of sinks that are not supported? If not we could say this supports all types of sync including integrated support for checkpointing as well as FileSystem support.
- does this mean that FileSystem is not a checkpointing sink? This is bit confusing as the docs talk of checkpointing FileSystemCheckpointStorage. I wonder if we could be clear to prevent misreadings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3093799751
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code, recommended for checkpointing.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code, has FileSystem sink support.
Review Comment:
I've made each bullet point more explicit about capabilities (checkpointing + FileSink support). A table could be a nice addition in a follow-up, but I'd prefer to keep this PR focused on the current scope. It's already been reviewed and approved by 3 reviewers. Happy to create a follow-up issue for a capabilities comparison table if you think it's valuable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3094208822
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers, removing the dependency from Hadoop entirely. Supports both checkpointing and the FileSink in a single plugin, removing the need to choose between Presto (checkpointing) and Hadoop (FileSink). [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code. The proven choice for checkpointing in production.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code. Supports both checkpointing and the FileSink.
+
+All three are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+
+## Common Configuration
+
+### Configure Access Credentials
+
+After setting up the S3 FileSystem implementation, you need to make sure that Flink is allowed to access your S3 buckets.
+
+#### Identity and Access Management (IAM) (Recommended)
+
+The recommended way of setting up credentials on AWS is via [Identity and Access Management (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html). You can use IAM features to securely give Flink instances the credentials that they need to access S3 buckets. Details about how to do this are beyond the scope of this documentation. Please refer to the AWS user guide. What you are looking for are [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+
+If you set this up correctly, you can manage access to S3 within AWS and don't need to distribute any access keys to Flink.
+
+#### Delegation Tokens
+
+[Delegation tokens]({{< ref "docs/deployment/security/security-delegation-token" >}}) provide time-bounded, automatically negotiated credentials. The JobManager uses long-lived credentials (access key and secret key) to call [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) and obtain short-lived session tokens, which are then automatically distributed to TaskManagers.
+
+Each S3 implementation has its own delegation token provider with a dedicated configuration prefix. You must set the `access-key`, `secret-key`, and `region` under the corresponding prefix for the implementation you are using:
+
+```yaml
+# For Native S3 implementation
+security.delegation.token.provider.s3-native.access-key: your-access-key
+security.delegation.token.provider.s3-native.secret-key: your-secret-key
+security.delegation.token.provider.s3-native.region: us-east-1
+
+# For Hadoop implementation
+security.delegation.token.provider.s3-hadoop.access-key: your-access-key
+security.delegation.token.provider.s3-hadoop.secret-key: your-secret-key
+security.delegation.token.provider.s3-hadoop.region: us-east-1
+
+# For Presto implementation
+security.delegation.token.provider.s3-presto.access-key: your-access-key
+security.delegation.token.provider.s3-presto.secret-key: your-secret-key
+security.delegation.token.provider.s3-presto.region: us-east-1
+```
+
+All three values (`access-key`, `secret-key`, `region`) must be set for delegation tokens to be issued. The `DynamicTemporaryAWSCredentialsProvider` is automatically included in the credentials provider chain for each implementation, so TaskManagers will consume the distributed tokens without additional configuration.
+
+#### Access Keys
+
+Access to S3 can be granted via your **access and secret key pair**. While access keys are not inherently insecure, IAM roles are preferred as they avoid the need to manage and distribute static credentials. See the [introduction of IAM roles](https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2) for more context.
+
+You need to configure both `s3.access-key` and `s3.secret-key` in Flink's [configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}):
+
+```yaml
+s3.access-key: your-access-key
+s3.secret-key: your-secret-key
+```
+
+### Configure Non-S3 Endpoint
+
+The S3 filesystems also support using S3 compliant object stores.
+To do so, configure your endpoint in [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}):
+
+```yaml
+s3.endpoint: your-endpoint-hostname
+```
+
+### Configure Path Style Access
+
+Some S3 compliant object stores might not have virtual host style addressing enabled by default. In such cases, you will have to provide the property to enable path style access in [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}):
+
+```yaml
+s3.path-style-access: true
+```
{{< hint info >}}
-You don't have to configure this manually if you are running [Flink on EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html).
+The legacy configuration key `s3.path.style.access` is still supported as a fallback for backward compatibility.
{{< /hint >}}
-Flink provides two file systems to talk to Amazon S3, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
-Both implementations are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+## Implementation Details
- - `flink-s3-fs-presto`, registered under the scheme *s3://* and *s3p://*, is based on code from the [Presto project](https://prestodb.io/).
- You can configure it using [the same configuration keys as the Presto file system](https://prestodb.io/docs/0.272/connector/hive.html#amazon-s3-configuration), by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). The Presto S3 implementation is the recommended file system for checkpointing to S3.
+### Native S3 FileSystem (Experimental)
- - `flink-s3-fs-hadoop`, registered under *s3://* and *s3a://*, based on code from the [Hadoop Project](https://hadoop.apache.org/).
- The file system can be [configured using Hadoop's s3a configuration keys](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A) by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}).
-
- For example, Hadoop has a `fs.s3a.connection.maximum` configuration key. If you want to change it, you need to put `s3.connection.maximum: xyz` to the [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). Flink will internally translate this back to `fs.s3a.connection.maximum`. There is no need to pass configuration parameters using Hadoop's XML configuration files.
-
- It is the only S3 file system with support for the [FileSystem]({{< ref "docs/connectors/datastream/filesystem" >}}).
-
+{{< hint warning >}}
+**Experimental**: The Native S3 FileSystem is experimental in Flink 2.3. It is functionally complete and has demonstrated strong performance in benchmarks.
+{{< /hint >}}
-Both `flink-s3-fs-hadoop` and `flink-s3-fs-presto` register default FileSystem
-wrappers for URIs with the *s3://* scheme, `flink-s3-fs-hadoop` also registers
-for *s3a://* and `flink-s3-fs-presto` also registers for *s3p://*, so you can
-use this to use both at the same time.
-For example, the job uses the [FileSystem]({{< ref "docs/connectors/datastream/filesystem" >}}) which only supports Hadoop, but uses Presto for checkpointing.
-In this case, you should explicitly use *s3a://* as a scheme for the sink (Hadoop) and *s3p://* for checkpointing (Presto).
+The Native S3 FileSystem is a pure-Java implementation built on the AWS SDK v2 completely removing the dependencies from hadoop. It is registered under the schemes *s3://* and *s3a://*. It requires no additional dependencies and provides a drop-in replacement for the Presto and Hadoop implementations.
-To use `flink-s3-fs-hadoop` or `flink-s3-fs-presto`, copy the respective JAR file from the `opt` directory to the `plugins` directory of your Flink distribution before starting Flink, e.g.
+#### Setup
+
+To use the Native S3 FileSystem, copy the JAR file from the `opt` directory to the `plugins` directory:
```bash
-mkdir ./plugins/s3-fs-presto
-cp ./opt/flink-s3-fs-presto-{{< version >}}.jar ./plugins/s3-fs-presto/
+mkdir -p ./plugins/s3-fs-native
+cp ./opt/flink-s3-fs-native-{{< version >}}.jar ./plugins/s3-fs-native/
```
-#### Configure Access Credentials
+#### Features
-After setting up the S3 FileSystem wrapper, you need to make sure that Flink is allowed to access your S3 buckets.
+- **No external dependencies**: Built on AWS SDK v2 with minimal footprint
+- **Drop-in replacement**: Compatible with the same S3 URI schemes (`s3://`, `s3a://`)
+- **FileSystem sink support**: Supports the [FileSystem sink]({{< ref "docs/connectors/datastream/filesystem" >}}) via `RecoverableWriter`
+- **Encryption support**: Server-side encryption (SSE-S3, SSE-KMS)
+- **Assume role**: Cross-account access via IAM role assumption
+- **Entropy injection**: Optimize S3 scalability through random key prefixes
+- **Bulk copy**: Efficient multi-part copy operations via S3TransferManager
-##### Identity and Access Management (IAM) (Recommended)
+#### Configuration
-The recommended way of setting up credentials on AWS is via [Identity and Access Management (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html). You can use IAM features to securely give Flink instances the credentials that they need to access S3 buckets. Details about how to do this are beyond the scope of this documentation. Please refer to the AWS user guide. What you are looking for are [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+The Native S3 FileSystem uses the following configuration options:
-If you set this up correctly, you can manage access to S3 within AWS and don't need to distribute any access keys to Flink.
+```yaml
+# AWS credentials (if using static credentials)
+s3.access-key: your-access-key
+s3.secret-key: your-secret-key
-##### Access Keys (Discouraged)
+# AWS region (optional; auto-detected if not specified)
+s3.region: us-east-1
-Access to S3 can be granted via your **access and secret key pair**. Please note that this is discouraged since the [introduction of IAM roles](https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2).
+# Custom S3 endpoint for S3-compatible storage
+s3.endpoint: your-endpoint-hostname
+
+# Path style access for S3-compatible storage
+s3.path-style-access: true
+
+# Server-side encryption
+s3.sse.type: sse-s3 # or sse-kms, aws:kms, AES256, none (default)
+s3.sse.kms.key-id: arn:aws:kms:region:account:key/id # For SSE-KMS
+
+# IAM role assumption for cross-account access
+s3.assume-role.arn: arn:aws:iam::account:role/RoleName
+s3.assume-role.external-id: external-id-if-required
+s3.assume-role.session-name: flink-s3-session
+s3.assume-role.session-duration: 3600
+
+# Performance tuning
+s3.upload.min.part.size: 5242880 # 5MB default
+s3.upload.max.concurrent.uploads: 4 # Based on CPU cores
+s3.read.buffer.size: 262144 # 256KB default
+s3.async.enabled: true # Enable async operations
+s3.bulk-copy.enabled: true # Enable bulk copy
+s3.bulk-copy.max-concurrent: 16 # Max concurrent copy ops
+
+# Entropy injection for scalability
+s3.entropy.key: _entropy_
+s3.entropy.length: 4
+
+# Retry configuration
+s3.retry.max-num-retries: 3
+
+# Credentials provider (optional; see note below)
+# fs.s3.aws.credentials.provider: software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider
+```
+
+When `fs.s3.aws.credentials.provider` is not set, the Native S3 FileSystem automatically builds a credentials chain in the following order: delegation tokens, static credentials (if `s3.access-key` and `s3.secret-key` are configured), and the AWS SDK v2 `DefaultCredentialsProvider` (environment variables, instance profiles, etc.). You only need to set this option if you require a custom provider chain.
+
+See the [AWS SDK v2 documentation](https://docs.aws.amazon.com/sdk-for-java/) for additional configuration details.
+
+---
-You need to configure both `s3.access-key` and `s3.secret-key` in Flink's [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}):
+### Presto S3 FileSystem
+
+{{< hint info >}}
+You don't have to configure this manually if you are running [Flink on EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html).
+{{< /hint >}}
+
+The Presto S3 FileSystem is based on code from the [Presto project](https://prestodb.io/). It is registered under the schemes *s3://* and *s3p://*.
+
+#### Features
+
+- **Recommended for checkpointing**: The Presto implementation is the recommended file system for checkpointing to S3
+- **Self-contained**: No Hadoop dependency required
+- **Production-ready**: Stable and widely used
+
+#### Setup
+
+To use the Presto S3 FileSystem, copy the JAR file from the `opt` directory to the `plugins` directory:
+
+```bash
+mkdir -p ./plugins/s3-fs-presto
+cp ./opt/flink-s3-fs-presto-{{< version >}}.jar ./plugins/s3-fs-presto/
+```
+
+#### Configuration
+
+Configure it using [the same configuration keys as the Presto file system](https://prestodb.io/docs/0.272/connector/hive.html#amazon-s3-configuration), by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}):
```yaml
+# AWS credentials
s3.access-key: your-access-key
s3.secret-key: your-secret-key
-```
-You can limit this configuration to JobManagers by using [Flink configuration file]({{< ref "docs/deployment/security/security-delegation-token" >}}).
+# Custom endpoint
+s3.endpoint: your-endpoint-hostname
-```yaml
-# flink-s3-fs-hadoop
-fs.s3a.aws.credentials.provider: org.apache.flink.fs.s3.common.token.DynamicTemporaryAWSCredentialsProvider
-# flink-s3-fs-presto
+# Path style access
+s3.path-style-access: true
+
+# Credentials provider
presto.s3.credentials-provider: org.apache.flink.fs.s3.common.token.DynamicTemporaryAWSCredentialsProvider
```
-## Configure Non-S3 Endpoint
+Refer to the [Presto documentation](https://prestodb.io/docs/0.272/connector/hive.html#amazon-s3-configuration) for all available configuration options.
-The S3 Filesystems also support using S3 compliant object stores such as [IBM's Cloud Object Storage](https://www.ibm.com/cloud/object-storage) and [MinIO](https://min.io/).
-To do so, configure your endpoint in [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}).
+---
-```yaml
-s3.endpoint: your-endpoint-hostname
+### Hadoop S3 FileSystem
+
+The Hadoop S3 FileSystem is based on code from the [Hadoop Project](https://hadoop.apache.org/). It is registered under the schemes *s3://* and *s3a://*.
Review Comment:
the links says "Presto uses its own S3 filesystem for the URI prefixes s3://, s3n:// and s3a://." missing s3n:// .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3093867384
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
Review Comment:
I've consolidated those two sentences into one.
"checkpointing" and "FileSink" refer to two distinct ways Flink uses the filesystem, not two types of sinks.
Historically, the Presto plugin was recommended for checkpointing (better streaming write performance) while the Hadoop plugin was needed for FileSink (it implements RecoverableWriter). The Native S3 implementation supports both in a single plugin.
This is not about sink types. it's about which S3 plugin you need for which Flink feature.
I've updated the text to make this clearer:
> "Supports both checkpointing and the FileSink in a single plugin, removing the need to choose between
> Presto (checkpointing) and Hadoop (FileSink)."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3094171049
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code, recommended for checkpointing.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code, has FileSystem sink support.
+
+All three are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+
+## Common Configuration
+
+### Configure Access Credentials
+
+After setting up the S3 FileSystem implementation, you need to make sure that Flink is allowed to access your S3 buckets.
+
+#### Identity and Access Management (IAM) (Recommended)
+
+The recommended way of setting up credentials on AWS is via [Identity and Access Management (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html). You can use IAM features to securely give Flink instances the credentials that they need to access S3 buckets. Details about how to do this are beyond the scope of this documentation. Please refer to the AWS user guide. What you are looking for are [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+
+If you set this up correctly, you can manage access to S3 within AWS and don't need to distribute any access keys to Flink.
+
+#### Delegation Tokens
+
+[Delegation tokens]({{< ref "docs/deployment/security/security-delegation-token" >}}) provide time-bounded, automatically negotiated credentials. The JobManager uses long-lived credentials to call AWS STS and obtain short-lived session tokens, which are then automatically distributed to TaskManagers.
+
+Each S3 implementation has its own delegation token provider with a dedicated configuration prefix. You must set the `access-key`, `secret-key`, and `region` under the corresponding prefix for the implementation you are using:
+
+```yaml
+# For Native S3 implementation
+security.delegation.token.provider.s3-native.access-key: your-access-key
+security.delegation.token.provider.s3-native.secret-key: your-secret-key
+security.delegation.token.provider.s3-native.region: us-east-1
+
+# For Hadoop implementation
+security.delegation.token.provider.s3-hadoop.access-key: your-access-key
+security.delegation.token.provider.s3-hadoop.secret-key: your-secret-key
+security.delegation.token.provider.s3-hadoop.region: us-east-1
+
+# For Presto implementation
+security.delegation.token.provider.s3-presto.access-key: your-access-key
+security.delegation.token.provider.s3-presto.secret-key: your-secret-key
+security.delegation.token.provider.s3-presto.region: us-east-1
+```
+
+All three values (`access-key`, `secret-key`, `region`) must be set for delegation tokens to be issued. The `DynamicTemporaryAWSCredentialsProvider` is automatically included in the credentials provider chain for each implementation, so TaskManagers will consume the distributed tokens without additional configuration.
+
+#### Access Keys
+
+Access to S3 can be granted via your **access and secret key pair**. While access keys are not inherently insecure, IAM roles are preferred as they avoid the need to manage and distribute static credentials. See the [introduction of IAM roles](https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2) for more context.
Review Comment:
putting this text into the docs I think would help clarify without going into too much detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3092141286
##########
docs/content.zh/docs/deployment/filesystems/s3.md:
##########
@@ -50,127 +48,321 @@ env.fromSource(
"s3-input"
);
-// 写入 S3 bucket
+// Write to S3 bucket
stream.sinkTo(
- FileSink.forRowFormat(
- new Path("s3://<bucket>/<endpoint>"), new SimpleStringEncoder<>()
- ).build()
+ FileSink.forRowFormat(
+ new Path("s3://<bucket>/<endpoint>"), new SimpleStringEncoder<>()
+ ).build()
);
-
-// 使用 S3 作为 checkpoint storage
+// Use S3 as checkpoint storage
Configuration config = new Configuration();
config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "s3://<your-bucket>/<endpoint>");
env.configure(config);
```
-注意这些例子并*不详尽*，S3 同样可以用在其他场景，包括 [JobManager 高可用配置]({{< ref "docs/deployment/ha/overview" >}}) 或 [RocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend)，以及所有 Flink 需要使用文件系统 URI 的位置。
+Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
Review Comment:
Thank you @spuru9, for the review. Even if I use AI, I won't know whether the word it will suggest is appropriate. I have no way to verify that deterministically.
Not sure what the right approach is, I have also posted about the same issue and am seeked for help in a Slack thread in #dev channel
https://apache-flink.slack.com/archives/C03GV7L3G2C/p1775930350489819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3092344795
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code, recommended for checkpointing.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code, has FileSystem sink support.
Review Comment:
maybe for completeness say whether this supports checkpointing. Could be have a table of all 3 connectors and their capabilities?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3092385061
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code, recommended for checkpointing.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code, has FileSystem sink support.
+
+All three are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+
+## Common Configuration
+
+### Configure Access Credentials
+
+After setting up the S3 FileSystem implementation, you need to make sure that Flink is allowed to access your S3 buckets.
+
+#### Identity and Access Management (IAM) (Recommended)
+
+The recommended way of setting up credentials on AWS is via [Identity and Access Management (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html). You can use IAM features to securely give Flink instances the credentials that they need to access S3 buckets. Details about how to do this are beyond the scope of this documentation. Please refer to the AWS user guide. What you are looking for are [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+
+If you set this up correctly, you can manage access to S3 within AWS and don't need to distribute any access keys to Flink.
+
+#### Delegation Tokens
+
+[Delegation tokens]({{< ref "docs/deployment/security/security-delegation-token" >}}) provide time-bounded, automatically negotiated credentials. The JobManager uses long-lived credentials to call AWS STS and obtain short-lived session tokens, which are then automatically distributed to TaskManagers.
Review Comment:
I think that it would be worth while detailing what the long lived tokens are, I assume these are the access-key and secret-key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3093791504
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code, recommended for checkpointing.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code, has FileSystem sink support.
+
+All three are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+
+## Common Configuration
+
+### Configure Access Credentials
+
+After setting up the S3 FileSystem implementation, you need to make sure that Flink is allowed to access your S3 buckets.
+
+#### Identity and Access Management (IAM) (Recommended)
+
+The recommended way of setting up credentials on AWS is via [Identity and Access Management (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html). You can use IAM features to securely give Flink instances the credentials that they need to access S3 buckets. Details about how to do this are beyond the scope of this documentation. Please refer to the AWS user guide. What you are looking for are [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+
+If you set this up correctly, you can manage access to S3 within AWS and don't need to distribute any access keys to Flink.
+
+#### Delegation Tokens
+
+[Delegation tokens]({{< ref "docs/deployment/security/security-delegation-token" >}}) provide time-bounded, automatically negotiated credentials. The JobManager uses long-lived credentials to call AWS STS and obtain short-lived session tokens, which are then automatically distributed to TaskManagers.
+
+Each S3 implementation has its own delegation token provider with a dedicated configuration prefix. You must set the `access-key`, `secret-key`, and `region` under the corresponding prefix for the implementation you are using:
+
+```yaml
+# For Native S3 implementation
+security.delegation.token.provider.s3-native.access-key: your-access-key
+security.delegation.token.provider.s3-native.secret-key: your-secret-key
+security.delegation.token.provider.s3-native.region: us-east-1
+
+# For Hadoop implementation
+security.delegation.token.provider.s3-hadoop.access-key: your-access-key
+security.delegation.token.provider.s3-hadoop.secret-key: your-secret-key
+security.delegation.token.provider.s3-hadoop.region: us-east-1
+
+# For Presto implementation
+security.delegation.token.provider.s3-presto.access-key: your-access-key
+security.delegation.token.provider.s3-presto.secret-key: your-secret-key
+security.delegation.token.provider.s3-presto.region: us-east-1
+```
+
+All three values (`access-key`, `secret-key`, `region`) must be set for delegation tokens to be issued. The `DynamicTemporaryAWSCredentialsProvider` is automatically included in the credentials provider chain for each implementation, so TaskManagers will consume the distributed tokens without additional configuration.
+
+#### Access Keys
+
+Access to S3 can be granted via your **access and secret key pair**. While access keys are not inherently insecure, IAM roles are preferred as they avoid the need to manage and distribute static credentials. See the [introduction of IAM roles](https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2) for more context.
Review Comment:
These are three independent credential approaches
1. IAM,
2. Delegation Tokens,
3. Access Keys
They are alternatives, not related steps. With IAM roles (e.g., EC2 instance profiles or EKS IRSA), Flink inherits credentials from the environment automatically, no s3.access-key/s3.secret-key needed. The Access Keys section is a separate option for environments where IAM roles aren't available. The linked blog post provides context on why IAM roles are preferred over static keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3098835851
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,208 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations:
-{{< hint info >}}
-You don't have to configure this manually if you are running [Flink on EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html).
-{{< /hint >}}
+| Implementation | Checkpointing | FileSink | Notes |
+|---------------|:---:|:---:|-------|
+| **Native S3** (`flink-s3-fs-native`) | ✓ | ✓ | **Experimental** in Flink 2.3. Built on AWS SDK v2; no Hadoop dependency. |
+| **Presto S3** (`flink-s3-fs-presto`) | ✓ | x | Production-proven for checkpointing. |
+| **Hadoop S3** (`flink-s3-fs-hadoop`) | ✓ | ✓ | Mature; the only stable implementation that provides `RecoverableWriter` for the FileSink. |
-Flink provides two file systems to talk to Amazon S3, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
-Both implementations are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+Previously, users had to choose between Presto (recommended for checkpointing throughput) and Hadoop (the only implementation with `RecoverableWriter`, required by the [FileSink]({{< ref "docs/connectors/datastream/filesystem" >}})). The Native S3 implementation unifies both capabilities in a single plugin and [benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to Presto at state sizes up to 15 GB.
Review Comment:
I understand the concern, but I think the current text is already appropriately qualified:
1. "benchmarks show" — attributes the claim to measured results, not an absolute guarantee
2. "~2x" and "~200 MB/s vs ~90 MB/s" — approximate figures, not exact promises
3. "at state sizes up to 15 GB" — explicitly scopes the conditions
4. The sentence links directly to the https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396 with the full benchmark methodology, so readers can assess for
themselves
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #27937:
URL: https://github.com/apache/flink/pull/27937#issuecomment-4266822286
@davidradl PTAL at the updated changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal merged PR #27937:
URL: https://github.com/apache/flink/pull/27937
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #27937:
URL: https://github.com/apache/flink/pull/27937#issuecomment-4272985595
Hi, @Samrat002 Could the patch use cherry-pick to keep the commit-id(of the main branch) ? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3092340942
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code, recommended for checkpointing.
Review Comment:
I am curiou why this is recommended for checkpointing when the new native version has checkpointing support. What are the pros and cons of implementing checkpointing between these 2 connectors and why is presto recommended?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3092385061
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code, recommended for checkpointing.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code, has FileSystem sink support.
+
+All three are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+
+## Common Configuration
+
+### Configure Access Credentials
+
+After setting up the S3 FileSystem implementation, you need to make sure that Flink is allowed to access your S3 buckets.
+
+#### Identity and Access Management (IAM) (Recommended)
+
+The recommended way of setting up credentials on AWS is via [Identity and Access Management (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html). You can use IAM features to securely give Flink instances the credentials that they need to access S3 buckets. Details about how to do this are beyond the scope of this documentation. Please refer to the AWS user guide. What you are looking for are [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+
+If you set this up correctly, you can manage access to S3 within AWS and don't need to distribute any access keys to Flink.
+
+#### Delegation Tokens
+
+[Delegation tokens]({{< ref "docs/deployment/security/security-delegation-token" >}}) provide time-bounded, automatically negotiated credentials. The JobManager uses long-lived credentials to call AWS STS and obtain short-lived session tokens, which are then automatically distributed to TaskManagers.
Review Comment:
I think that it would be worth while detailing what the long lived tokens are, I assume these are the access-key and secret-key. And I assume the delegation tokens are the short lived token.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3098713035
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,208 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations:
-{{< hint info >}}
-You don't have to configure this manually if you are running [Flink on EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html).
-{{< /hint >}}
+| Implementation | Checkpointing | FileSink | Notes |
+|---------------|:---:|:---:|-------|
+| **Native S3** (`flink-s3-fs-native`) | ✓ | ✓ | **Experimental** in Flink 2.3. Built on AWS SDK v2; no Hadoop dependency. |
+| **Presto S3** (`flink-s3-fs-presto`) | ✓ | x | Production-proven for checkpointing. |
+| **Hadoop S3** (`flink-s3-fs-hadoop`) | ✓ | ✓ | Mature; the only stable implementation that provides `RecoverableWriter` for the FileSink. |
-Flink provides two file systems to talk to Amazon S3, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
-Both implementations are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+Previously, users had to choose between Presto (recommended for checkpointing throughput) and Hadoop (the only implementation with `RecoverableWriter`, required by the [FileSink]({{< ref "docs/connectors/datastream/filesystem" >}})). The Native S3 implementation unifies both capabilities in a single plugin and [benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to Presto at state sizes up to 15 GB.
Review Comment:
I suggest if mentioning benchmarks you need to be careful to say this is not the truth in all situations . We could say something like we have shown in our environment ... To save embarrassment if a particular scenario is non performant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3098874309
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,208 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations:
-{{< hint info >}}
-You don't have to configure this manually if you are running [Flink on EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html).
-{{< /hint >}}
+| Implementation | Checkpointing | FileSink | Notes |
+|---------------|:---:|:---:|-------|
+| **Native S3** (`flink-s3-fs-native`) | ✓ | ✓ | **Experimental** in Flink 2.3. Built on AWS SDK v2; no Hadoop dependency. |
+| **Presto S3** (`flink-s3-fs-presto`) | ✓ | x | Production-proven for checkpointing. |
+| **Hadoop S3** (`flink-s3-fs-hadoop`) | ✓ | ✓ | Mature; the only stable implementation that provides `RecoverableWriter` for the FileSink. |
-Flink provides two file systems to talk to Amazon S3, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
-Both implementations are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+Previously, users had to choose between Presto (recommended for checkpointing throughput) and Hadoop (the only implementation with `RecoverableWriter`, required by the [FileSink]({{< ref "docs/connectors/datastream/filesystem" >}})). The Native S3 implementation unifies both capabilities in a single plugin and [benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to Presto at state sizes up to 15 GB.
Review Comment:
If we still want to propagate the exact methodology and numbers we can create a blog post about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3092210596
##########
docs/content.zh/docs/deployment/filesystems/s3.md:
##########
@@ -50,127 +48,321 @@ env.fromSource(
"s3-input"
);
-// 写入 S3 bucket
+// Write to S3 bucket
stream.sinkTo(
- FileSink.forRowFormat(
- new Path("s3://<bucket>/<endpoint>"), new SimpleStringEncoder<>()
- ).build()
+ FileSink.forRowFormat(
+ new Path("s3://<bucket>/<endpoint>"), new SimpleStringEncoder<>()
+ ).build()
);
-
-// 使用 S3 作为 checkpoint storage
+// Use S3 as checkpoint storage
Configuration config = new Configuration();
config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "s3://<your-bucket>/<endpoint>");
env.configure(config);
```
-注意这些例子并*不详尽*，S3 同样可以用在其他场景，包括 [JobManager 高可用配置]({{< ref "docs/deployment/ha/overview" >}}) 或 [RocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend)，以及所有 Flink 需要使用文件系统 URI 的位置。
+Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
Review Comment:
Ok. Got it.
Maybe by then we can add a TODO comment, if thats blocking you. Just so it comes up in future when someone pick it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3094155326
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers, removing the dependency from Hadoop entirely. Supports both checkpointing and the FileSink in a single plugin, removing the need to choose between Presto (checkpointing) and Hadoop (FileSink). [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code. The proven choice for checkpointing in production.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code. Supports both checkpointing and the FileSink.
Review Comment:
Am I missing something, you say` Hadoop (FileSink)` but then say `Based on Hadoop project code. Supports both checkpointing and the FileSink.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3098442128
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers, removing the dependency from Hadoop entirely. Supports both checkpointing and the FileSink in a single plugin, removing the need to choose between Presto (checkpointing) and Hadoop (FileSink). [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code. The proven choice for checkpointing in production.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code. Supports both checkpointing and the FileSink.
Review Comment:
You're right, this reads as contradictory, and I should have caught it earlier.
The historical situation is:
```
- Presto: supports checkpointing only. Calling createRecoverableWriter() throws UnsupportedOperationException (see PrestoS3RecoverableWriterTest).
- Hadoop: supports both checkpointing and the FileSink (it provides the RecoverableWriter implementation).
```
So saying "Hadoop (FileSink)" in the Native bullet was misleading — it implied Hadoop only does FileSink, when it actually does both. The real distinction is that Presto was recommended for checkpointing throughput, and Hadoop was the only option for FileSink since Presto doesn't implement RecoverableWriter at all.
I've reworked this section into a capabilities table that makes each implementation's support explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3098855937
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,208 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations:
-{{< hint info >}}
-You don't have to configure this manually if you are running [Flink on EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html).
-{{< /hint >}}
+| Implementation | Checkpointing | FileSink | Notes |
+|---------------|:---:|:---:|-------|
+| **Native S3** (`flink-s3-fs-native`) | ✓ | ✓ | **Experimental** in Flink 2.3. Built on AWS SDK v2; no Hadoop dependency. |
+| **Presto S3** (`flink-s3-fs-presto`) | ✓ | x | Production-proven for checkpointing. |
+| **Hadoop S3** (`flink-s3-fs-hadoop`) | ✓ | ✓ | Mature; the only stable implementation that provides `RecoverableWriter` for the FileSink. |
Review Comment:
Thanks for the suggestion @davidradl. I thought about this, but I think RecoverableWriter is the wrong abstraction level for this table. it's an internal API, not a user-facing capability. A separate column would be redundant with the FileSink column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #27937:
URL: https://github.com/apache/flink/pull/27937#issuecomment-4257094122
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f309c80ece708e4b4e47de53358f4fb8b46e9221",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f309c80ece708e4b4e47de53358f4fb8b46e9221",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f309c80ece708e4b4e47de53358f4fb8b46e9221 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3092210596
##########
docs/content.zh/docs/deployment/filesystems/s3.md:
##########
@@ -50,127 +48,321 @@ env.fromSource(
"s3-input"
);
-// 写入 S3 bucket
+// Write to S3 bucket
stream.sinkTo(
- FileSink.forRowFormat(
- new Path("s3://<bucket>/<endpoint>"), new SimpleStringEncoder<>()
- ).build()
+ FileSink.forRowFormat(
+ new Path("s3://<bucket>/<endpoint>"), new SimpleStringEncoder<>()
+ ).build()
);
-
-// 使用 S3 作为 checkpoint storage
+// Use S3 as checkpoint storage
Configuration config = new Configuration();
config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "s3://<your-bucket>/<endpoint>");
env.configure(config);
```
-注意这些例子并*不详尽*，S3 同样可以用在其他场景，包括 [JobManager 高可用配置]({{< ref "docs/deployment/ha/overview" >}}) 或 [RocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend)，以及所有 Flink 需要使用文件系统 URI 的位置。
+Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
Review Comment:
Ok. Got it.
Maybe by then we can add a TODO comment, would be nice. Just so it comes up in future when someone pick it up. Nothing major though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3092415380
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code, recommended for checkpointing.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code, has FileSystem sink support.
+
+All three are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+
+## Common Configuration
+
+### Configure Access Credentials
+
+After setting up the S3 FileSystem implementation, you need to make sure that Flink is allowed to access your S3 buckets.
+
+#### Identity and Access Management (IAM) (Recommended)
+
+The recommended way of setting up credentials on AWS is via [Identity and Access Management (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html). You can use IAM features to securely give Flink instances the credentials that they need to access S3 buckets. Details about how to do this are beyond the scope of this documentation. Please refer to the AWS user guide. What you are looking for are [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+
+If you set this up correctly, you can manage access to S3 within AWS and don't need to distribute any access keys to Flink.
+
+#### Delegation Tokens
+
+[Delegation tokens]({{< ref "docs/deployment/security/security-delegation-token" >}}) provide time-bounded, automatically negotiated credentials. The JobManager uses long-lived credentials to call AWS STS and obtain short-lived session tokens, which are then automatically distributed to TaskManagers.
+
+Each S3 implementation has its own delegation token provider with a dedicated configuration prefix. You must set the `access-key`, `secret-key`, and `region` under the corresponding prefix for the implementation you are using:
+
+```yaml
+# For Native S3 implementation
+security.delegation.token.provider.s3-native.access-key: your-access-key
+security.delegation.token.provider.s3-native.secret-key: your-secret-key
+security.delegation.token.provider.s3-native.region: us-east-1
+
+# For Hadoop implementation
+security.delegation.token.provider.s3-hadoop.access-key: your-access-key
+security.delegation.token.provider.s3-hadoop.secret-key: your-secret-key
+security.delegation.token.provider.s3-hadoop.region: us-east-1
+
+# For Presto implementation
+security.delegation.token.provider.s3-presto.access-key: your-access-key
+security.delegation.token.provider.s3-presto.secret-key: your-secret-key
+security.delegation.token.provider.s3-presto.region: us-east-1
+```
+
+All three values (`access-key`, `secret-key`, `region`) must be set for delegation tokens to be issued. The `DynamicTemporaryAWSCredentialsProvider` is automatically included in the credentials provider chain for each implementation, so TaskManagers will consume the distributed tokens without additional configuration.
+
+#### Access Keys
+
+Access to S3 can be granted via your **access and secret key pair**. While access keys are not inherently insecure, IAM roles are preferred as they avoid the need to manage and distribute static credentials. See the [introduction of IAM roles](https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2) for more context.
Review Comment:
Looking at the link it says "When you call S3 this way, the temporary security credentials that describe which EC2 instance has access and permission are retrieved automatically. " I am unsure how Flink associates itself with a role. I see we are asked to define `s3.access-key` and `s3.secret-key`, I am not sure how these relate to the role or the previous access keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3093820133
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers removing the dependency from hadoop entirely. This implementation supports both checkpointing and the FileSystem sink. The Native S3 FileSystem aims to provide integrated support for checkpointing as well as FileSystem sink, removing the need to use Presto S3 FileSystem for checkpointing and Hadoop S3 FileSystem for the FileSystem sink. [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code, recommended for checkpointing.
Review Comment:
Good question. The Native S3 implementation is Experimental in Flink 2.3 (as noted in the bullet point above). While benchmarks show it delivers ~2x higher checkpoint throughput, it hasn't had the same level of production hardening as Presto, which has been the go-to for checkpointing across many Flink deployments for years.
For users who need maximum reliability today, Presto is the safer bet. For users willing to adopt an experimental feature for the performance gains, Native is the way forward and the long-term goal is for Native to replace both Presto and Hadoop.
I've updated the Presto line to make this reasoning explicit: "The proven choice for checkpointing in production." which clarifies that the recommendation is about production maturity, not technical capability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3094193585
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,288 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations, each with different trade-offs:
+
+- **Native S3 FileSystem** (`flink-s3-fs-native`): Built directly on AWS SDK v2 with async I/O and parallel transfers, removing the dependency from Hadoop entirely. Supports both checkpointing and the FileSink in a single plugin, removing the need to choose between Presto (checkpointing) and Hadoop (FileSink). [Benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to the Presto implementation at state sizes up to 15 GB. **Experimental** in Flink 2.3.
+- **Presto S3 FileSystem** (`flink-s3-fs-presto`): Based on Presto project code. The proven choice for checkpointing in production.
+- **Hadoop S3 FileSystem** (`flink-s3-fs-hadoop`): Based on Hadoop project code. Supports both checkpointing and the FileSink.
+
+All three are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+
+## Common Configuration
+
+### Configure Access Credentials
+
+After setting up the S3 FileSystem implementation, you need to make sure that Flink is allowed to access your S3 buckets.
+
+#### Identity and Access Management (IAM) (Recommended)
+
+The recommended way of setting up credentials on AWS is via [Identity and Access Management (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html). You can use IAM features to securely give Flink instances the credentials that they need to access S3 buckets. Details about how to do this are beyond the scope of this documentation. Please refer to the AWS user guide. What you are looking for are [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+
+If you set this up correctly, you can manage access to S3 within AWS and don't need to distribute any access keys to Flink.
+
+#### Delegation Tokens
+
+[Delegation tokens]({{< ref "docs/deployment/security/security-delegation-token" >}}) provide time-bounded, automatically negotiated credentials. The JobManager uses long-lived credentials (access key and secret key) to call [AWS STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) and obtain short-lived session tokens, which are then automatically distributed to TaskManagers.
+
+Each S3 implementation has its own delegation token provider with a dedicated configuration prefix. You must set the `access-key`, `secret-key`, and `region` under the corresponding prefix for the implementation you are using:
+
+```yaml
+# For Native S3 implementation
+security.delegation.token.provider.s3-native.access-key: your-access-key
+security.delegation.token.provider.s3-native.secret-key: your-secret-key
+security.delegation.token.provider.s3-native.region: us-east-1
+
+# For Hadoop implementation
+security.delegation.token.provider.s3-hadoop.access-key: your-access-key
+security.delegation.token.provider.s3-hadoop.secret-key: your-secret-key
+security.delegation.token.provider.s3-hadoop.region: us-east-1
+
+# For Presto implementation
+security.delegation.token.provider.s3-presto.access-key: your-access-key
+security.delegation.token.provider.s3-presto.secret-key: your-secret-key
+security.delegation.token.provider.s3-presto.region: us-east-1
+```
+
+All three values (`access-key`, `secret-key`, `region`) must be set for delegation tokens to be issued. The `DynamicTemporaryAWSCredentialsProvider` is automatically included in the credentials provider chain for each implementation, so TaskManagers will consume the distributed tokens without additional configuration.
+
+#### Access Keys
+
+Access to S3 can be granted via your **access and secret key pair**. While access keys are not inherently insecure, IAM roles are preferred as they avoid the need to manage and distribute static credentials. See the [introduction of IAM roles](https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2) for more context.
+
+You need to configure both `s3.access-key` and `s3.secret-key` in Flink's [configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}):
+
+```yaml
+s3.access-key: your-access-key
+s3.secret-key: your-secret-key
+```
+
+### Configure Non-S3 Endpoint
+
+The S3 filesystems also support using S3 compliant object stores.
+To do so, configure your endpoint in [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}):
+
+```yaml
+s3.endpoint: your-endpoint-hostname
+```
+
+### Configure Path Style Access
+
+Some S3 compliant object stores might not have virtual host style addressing enabled by default. In such cases, you will have to provide the property to enable path style access in [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}):
+
+```yaml
+s3.path-style-access: true
+```
{{< hint info >}}
-You don't have to configure this manually if you are running [Flink on EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html).
+The legacy configuration key `s3.path.style.access` is still supported as a fallback for backward compatibility.
{{< /hint >}}
-Flink provides two file systems to talk to Amazon S3, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
-Both implementations are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+## Implementation Details
- - `flink-s3-fs-presto`, registered under the scheme *s3://* and *s3p://*, is based on code from the [Presto project](https://prestodb.io/).
- You can configure it using [the same configuration keys as the Presto file system](https://prestodb.io/docs/0.272/connector/hive.html#amazon-s3-configuration), by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). The Presto S3 implementation is the recommended file system for checkpointing to S3.
+### Native S3 FileSystem (Experimental)
- - `flink-s3-fs-hadoop`, registered under *s3://* and *s3a://*, based on code from the [Hadoop Project](https://hadoop.apache.org/).
- The file system can be [configured using Hadoop's s3a configuration keys](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A) by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}).
-
- For example, Hadoop has a `fs.s3a.connection.maximum` configuration key. If you want to change it, you need to put `s3.connection.maximum: xyz` to the [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). Flink will internally translate this back to `fs.s3a.connection.maximum`. There is no need to pass configuration parameters using Hadoop's XML configuration files.
-
- It is the only S3 file system with support for the [FileSystem]({{< ref "docs/connectors/datastream/filesystem" >}}).
-
+{{< hint warning >}}
+**Experimental**: The Native S3 FileSystem is experimental in Flink 2.3. It is functionally complete and has demonstrated strong performance in benchmarks.
+{{< /hint >}}
-Both `flink-s3-fs-hadoop` and `flink-s3-fs-presto` register default FileSystem
-wrappers for URIs with the *s3://* scheme, `flink-s3-fs-hadoop` also registers
-for *s3a://* and `flink-s3-fs-presto` also registers for *s3p://*, so you can
-use this to use both at the same time.
-For example, the job uses the [FileSystem]({{< ref "docs/connectors/datastream/filesystem" >}}) which only supports Hadoop, but uses Presto for checkpointing.
-In this case, you should explicitly use *s3a://* as a scheme for the sink (Hadoop) and *s3p://* for checkpointing (Presto).
+The Native S3 FileSystem is a pure-Java implementation built on the AWS SDK v2 completely removing the dependencies from hadoop. It is registered under the schemes *s3://* and *s3a://*. It requires no additional dependencies and provides a drop-in replacement for the Presto and Hadoop implementations.
-To use `flink-s3-fs-hadoop` or `flink-s3-fs-presto`, copy the respective JAR file from the `opt` directory to the `plugins` directory of your Flink distribution before starting Flink, e.g.
+#### Setup
+
+To use the Native S3 FileSystem, copy the JAR file from the `opt` directory to the `plugins` directory:
```bash
-mkdir ./plugins/s3-fs-presto
-cp ./opt/flink-s3-fs-presto-{{< version >}}.jar ./plugins/s3-fs-presto/
+mkdir -p ./plugins/s3-fs-native
+cp ./opt/flink-s3-fs-native-{{< version >}}.jar ./plugins/s3-fs-native/
```
-#### Configure Access Credentials
+#### Features
-After setting up the S3 FileSystem wrapper, you need to make sure that Flink is allowed to access your S3 buckets.
+- **No external dependencies**: Built on AWS SDK v2 with minimal footprint
+- **Drop-in replacement**: Compatible with the same S3 URI schemes (`s3://`, `s3a://`)
+- **FileSystem sink support**: Supports the [FileSystem sink]({{< ref "docs/connectors/datastream/filesystem" >}}) via `RecoverableWriter`
+- **Encryption support**: Server-side encryption (SSE-S3, SSE-KMS)
+- **Assume role**: Cross-account access via IAM role assumption
+- **Entropy injection**: Optimize S3 scalability through random key prefixes
+- **Bulk copy**: Efficient multi-part copy operations via S3TransferManager
-##### Identity and Access Management (IAM) (Recommended)
+#### Configuration
-The recommended way of setting up credentials on AWS is via [Identity and Access Management (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html). You can use IAM features to securely give Flink instances the credentials that they need to access S3 buckets. Details about how to do this are beyond the scope of this documentation. Please refer to the AWS user guide. What you are looking for are [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+The Native S3 FileSystem uses the following configuration options:
-If you set this up correctly, you can manage access to S3 within AWS and don't need to distribute any access keys to Flink.
+```yaml
+# AWS credentials (if using static credentials)
+s3.access-key: your-access-key
+s3.secret-key: your-secret-key
-##### Access Keys (Discouraged)
+# AWS region (optional; auto-detected if not specified)
+s3.region: us-east-1
-Access to S3 can be granted via your **access and secret key pair**. Please note that this is discouraged since the [introduction of IAM roles](https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2).
+# Custom S3 endpoint for S3-compatible storage
+s3.endpoint: your-endpoint-hostname
+
+# Path style access for S3-compatible storage
+s3.path-style-access: true
+
+# Server-side encryption
+s3.sse.type: sse-s3 # or sse-kms, aws:kms, AES256, none (default)
+s3.sse.kms.key-id: arn:aws:kms:region:account:key/id # For SSE-KMS
+
+# IAM role assumption for cross-account access
+s3.assume-role.arn: arn:aws:iam::account:role/RoleName
+s3.assume-role.external-id: external-id-if-required
+s3.assume-role.session-name: flink-s3-session
+s3.assume-role.session-duration: 3600
+
+# Performance tuning
+s3.upload.min.part.size: 5242880 # 5MB default
+s3.upload.max.concurrent.uploads: 4 # Based on CPU cores
+s3.read.buffer.size: 262144 # 256KB default
+s3.async.enabled: true # Enable async operations
+s3.bulk-copy.enabled: true # Enable bulk copy
+s3.bulk-copy.max-concurrent: 16 # Max concurrent copy ops
+
+# Entropy injection for scalability
+s3.entropy.key: _entropy_
+s3.entropy.length: 4
+
+# Retry configuration
+s3.retry.max-num-retries: 3
+
+# Credentials provider (optional; see note below)
+# fs.s3.aws.credentials.provider: software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider
+```
+
+When `fs.s3.aws.credentials.provider` is not set, the Native S3 FileSystem automatically builds a credentials chain in the following order: delegation tokens, static credentials (if `s3.access-key` and `s3.secret-key` are configured), and the AWS SDK v2 `DefaultCredentialsProvider` (environment variables, instance profiles, etc.). You only need to set this option if you require a custom provider chain.
+
+See the [AWS SDK v2 documentation](https://docs.aws.amazon.com/sdk-for-java/) for additional configuration details.
+
+---
-You need to configure both `s3.access-key` and `s3.secret-key` in Flink's [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}):
+### Presto S3 FileSystem
+
+{{< hint info >}}
+You don't have to configure this manually if you are running [Flink on EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html).
+{{< /hint >}}
+
+The Presto S3 FileSystem is based on code from the [Presto project](https://prestodb.io/). It is registered under the schemes *s3://* and *s3p://*.
+
+#### Features
+
+- **Recommended for checkpointing**: The Presto implementation is the recommended file system for checkpointing to S3
+- **Self-contained**: No Hadoop dependency required
+- **Production-ready**: Stable and widely used
+
+#### Setup
+
+To use the Presto S3 FileSystem, copy the JAR file from the `opt` directory to the `plugins` directory:
+
+```bash
+mkdir -p ./plugins/s3-fs-presto
+cp ./opt/flink-s3-fs-presto-{{< version >}}.jar ./plugins/s3-fs-presto/
+```
+
+#### Configuration
+
+Configure it using [the same configuration keys as the Presto file system](https://prestodb.io/docs/0.272/connector/hive.html#amazon-s3-configuration), by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}):
```yaml
+# AWS credentials
s3.access-key: your-access-key
s3.secret-key: your-secret-key
-```
-You can limit this configuration to JobManagers by using [Flink configuration file]({{< ref "docs/deployment/security/security-delegation-token" >}}).
+# Custom endpoint
+s3.endpoint: your-endpoint-hostname
-```yaml
-# flink-s3-fs-hadoop
-fs.s3a.aws.credentials.provider: org.apache.flink.fs.s3.common.token.DynamicTemporaryAWSCredentialsProvider
-# flink-s3-fs-presto
+# Path style access
+s3.path-style-access: true
+
+# Credentials provider
presto.s3.credentials-provider: org.apache.flink.fs.s3.common.token.DynamicTemporaryAWSCredentialsProvider
```
-## Configure Non-S3 Endpoint
+Refer to the [Presto documentation](https://prestodb.io/docs/0.272/connector/hive.html#amazon-s3-configuration) for all available configuration options.
Review Comment:
isnt this a a duplicate of line 253. I suggest removing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39118][docs] Add documentation for Native s3 FileSystem [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #27937:
URL: https://github.com/apache/flink/pull/27937#discussion_r3098922925
##########
docs/content/docs/deployment/filesystems/s3.md:
##########
@@ -64,94 +64,208 @@ env.configure(config);
Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{< ref "docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref "docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI (unless otherwise stated).
-For most use cases, you may use one of our `flink-s3-fs-hadoop` and `flink-s3-fs-presto` S3 filesystem plugins which are self-contained and easy to set up.
-For some cases, however, e.g., for using S3 as YARN's resource storage dir, it may be necessary to set up a specific Hadoop S3 filesystem implementation.
+## S3 FileSystem Implementations
-### Hadoop/Presto S3 File Systems plugins
+Flink provides three independent S3 filesystem implementations:
-{{< hint info >}}
-You don't have to configure this manually if you are running [Flink on EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html).
-{{< /hint >}}
+| Implementation | Checkpointing | FileSink | Notes |
+|---------------|:---:|:---:|-------|
+| **Native S3** (`flink-s3-fs-native`) | ✓ | ✓ | **Experimental** in Flink 2.3. Built on AWS SDK v2; no Hadoop dependency. |
+| **Presto S3** (`flink-s3-fs-presto`) | ✓ | x | Production-proven for checkpointing. |
+| **Hadoop S3** (`flink-s3-fs-hadoop`) | ✓ | ✓ | Mature; the only stable implementation that provides `RecoverableWriter` for the FileSink. |
-Flink provides two file systems to talk to Amazon S3, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
-Both implementations are self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use them.
+Previously, users had to choose between Presto (recommended for checkpointing throughput) and Hadoop (the only implementation with `RecoverableWriter`, required by the [FileSink]({{< ref "docs/connectors/datastream/filesystem" >}})). The Native S3 implementation unifies both capabilities in a single plugin and [benchmarks](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396) show ~2x higher checkpoint throughput (~200 MB/s vs ~90 MB/s) compared to Presto at state sizes up to 15 GB.
Review Comment:
> Maybe we can be less exact on numbers since it can change rapidly when we add new features or users have difference use-case. We can mention that measurements show performance gain and stability without exact details. This doc is for sales and how to use. If users are interested in the details (majority don't) they can find it manually without links.
make sense. i will update the doc to show performance gain but avoid exact numbers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org