You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Balaji Varadarajan (Jira)" <ji...@apache.org> on 2019/09/30 22:50:00 UTC
[jira] [Closed] (HUDI-269) Provide ability to throttle
DeltaStreamer sync runs
[ https://issues.apache.org/jira/browse/HUDI-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Balaji Varadarajan closed HUDI-269.
-----------------------------------
> Provide ability to throttle DeltaStreamer sync runs
> ---------------------------------------------------
>
> Key: HUDI-269
> URL: https://issues.apache.org/jira/browse/HUDI-269
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: deltastreamer
> Reporter: Balaji Varadarajan
> Assignee: Xing Pan
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: hudi_request_test.tar.gz, image-2019-09-25-08-51-19-686.png, image-2019-09-26-09-02-24-761.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Copied from [https://github.com/apache/incubator-hudi/issues/922]
> In some scenario in our cluster, we may want delta streamer to slow down a bit.
> so it's nice to have a parameter to control the min sync interval of each sync in continuous mode.
> this param is default to 0, so this should not affect current logic.
> minor pr: [#921|https://github.com/apache/incubator-hudi/pull/921]
> the main reason we want to slow it down is that aws s3 is charged by s3 get/put/list requests. we don't want to pay for too many requests for a really slow change table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)