You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Etienne CARRIERE (JIRA)" <ji...@apache.org> on 2018/06/11 13:42:00 UTC

[jira] [Updated] (FLINK-9560) RateLimiting for FileSystem

     [ https://issues.apache.org/jira/browse/FLINK-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Etienne CARRIERE updated FLINK-9560:
------------------------------------
    Description: 
*Pain*: On our system, we see that during checkpoint , all the bandwidth is take to send the checkpoint to object storage (s3 in our case)

*Proposal* : After the creation of some limitation on Filesystem (mostly number of connections with the  tickets FLINK-8125/FLINK-8198/FLINK-9468), I propose to add ratelimiting "per Filesystem" .

Proposal of implementation : Modify LimitedConnectionsFileSystem to add a ratelimiter on both Input and OutputStream.

 

  was:
*Pain*: On our system, we see that during checkpoint , all the bandwidth is take to send the checkpoint to object storage (s3 in our case)

*Proposal* : After the creation of some limitation on Filesystem (mostly number of connections with the  tickets FLINK-8125/FLINK-8198/FLINK-9468), I propose to add ratelimiting "per Filesystem" .

Proposal of implementation : Modify LimitedConnectionsFileSystem to add a ratelimiter on both Input and OutputStream.

Current issue : I would like to use the guava RateLimiter that is a good ratelimiter but it is guava and so not included in flink (dependency clash with hadoop) if I am right. What would be a right strategy in this case ?

 


> RateLimiting for FileSystem
> ---------------------------
>
>                 Key: FLINK-9560
>                 URL: https://issues.apache.org/jira/browse/FLINK-9560
>             Project: Flink
>          Issue Type: Improvement
>          Components: FileSystem
>    Affects Versions: 1.5.0
>            Reporter: Etienne CARRIERE
>            Priority: Major
>
> *Pain*: On our system, we see that during checkpoint , all the bandwidth is take to send the checkpoint to object storage (s3 in our case)
> *Proposal* : After the creation of some limitation on Filesystem (mostly number of connections with the  tickets FLINK-8125/FLINK-8198/FLINK-9468), I propose to add ratelimiting "per Filesystem" .
> Proposal of implementation : Modify LimitedConnectionsFileSystem to add a ratelimiter on both Input and OutputStream.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)