[jira] [Commented] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

["Ahmed Hamdy (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715483#comment-17715483 ] 

Ahmed Hamdy commented on FLINK-31872:
-------------------------------------

CC [~danny.cranmer] 

> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
> ------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-31872
>                 URL: https://issues.apache.org/jira/browse/FLINK-31872
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kinesis
>            Reporter: Ahmed Hamdy
>            Assignee: Ahmed Hamdy
>            Priority: Major
>             Fix For: aws-connector-4.2.0
>
>
> h1. Issue
> As part of FLINK-31772
> I performed a complete benchmark for {{KinesisStreamsSink}} after configuring rate limiting strategy.
> It appears that optimum values for rate limiting strategy parameters are dependent on use case (shard number/ parallellism/ record thouroughput)
> We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with one used for TCP congestion control but since parameters are use case dependent we would like to allow sink users to adjust parameters as suitable.
> h2. Requirements
>  - we *must* allow users to configure increment rate and decrease factor of AIMDRateLimitingStrategy for {{KinesisStreamsSink}}
>  - we *must* provide backward compatible default values identical to current values to introduce no further regressions.
> h2. Appendix
> h3. Performace Benchmark Results
> |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|Async sink/ Maximum Thourouput|% of Improvement|
> |Low/Low/Low|1|1|1024|10000|0.991|1|1|0.9|
> |Low/Low/High|1|1|102400|100|0.9943|1|1|0.57|
> |Low/Med/Low|1|8|1024|80000|4.12|4.57|0.57125|5.625|
> |Low/Med/High|1|8|102400|800|4.35|7.65|0.95625|41.25|
> |Med/Low/Low|8|1|1024|20000|0.852|0.846|0.846|-0.6|
> |Med/Low/High|8|1|102400|200|0.921|0.867|0.867|-5.4|
> |Med/Med/Low|8|8|1024|80000|5.37|4.76|0.595|-7.625|
> |Med/Med/High|8|8|102400|800|7.53|7.69|0.96125|2|
> |Med/High/Low|8|64|1024|80000|32.5|37.4|0.58438|7.65625|
> |Med/High/High|8|64|102400|800|47.27|60.4|0.94375|20.51562|
> |High/High/Low|256|256|1024|300000|127|127|0.49609|0|
> |High/High/High|256|256|102400|3000|225|246|0.96094|8.20313|



--
This message was sent by Atlassian Jira
(v8.20.10#820010)