You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Robert Metzger (Jira)" <ji...@apache.org> on 2021/09/28 09:13:00 UTC

[jira] [Created] (FLINK-24392) Upgrade presto s3 fs implementation to Trinio >= 348

Robert Metzger created FLINK-24392:
--------------------------------------

             Summary: Upgrade presto s3 fs implementation to Trinio >= 348
                 Key: FLINK-24392
                 URL: https://issues.apache.org/jira/browse/FLINK-24392
             Project: Flink
          Issue Type: Improvement
          Components: FileSystems
    Affects Versions: 1.14.0
            Reporter: Robert Metzger
             Fix For: 1.15.0


The Presto s3 filesystem implementation currently shipped with Flink doesn't support streaming uploads. All data needs to be materialized to a single file on disk, before it can be uploaded.
This can lead to situations where TaskManagers are running out of disk when creating a savepoint.

The Hadoop filesystem implementation supports streaming uploads (by using multipart uploads of smaller (say 100mb) files locally), but it does more API calls, leading to other issues.

Trinion 348 supports streaming uploads.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)