[jira] [Updated] (FLINK-25793) Kinesis Data Streams sink is not limiting throughput to the destination and therefore exceeding rate limits

["Zichen Liu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-25793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zichen Liu updated FLINK-25793:
-------------------------------
    Description: 
h2. Bug:

Kinesis Data Streams sink is not limiting throughput to the destination and therefore exceeding rate limits

*Cause:*

We are not throttling our requests downstream at all.

We should monitor for requests that have failed with ThroughputExceeded exceptions and reduce the throughput accordingly.

  was:
h2. Bug:

Async Sink Base is too being flushed too frequently resulting in backpressure even when buffer is near empty

*Cause:*

During a write(), flushIfAble() is called, which checks if the number of buffered elements is greater than a batch size, and if so, insists that the sink flushes immediately, even if the number of inFlightRequests is greater than the maximum allowed number of inFlightRequests, resulting in a yield of the current mailbox thread, and hence blocks.

Notice that this can occur even if the buffer is near empty, so the blocking behaviour is unnecessary and undesirable, since we would like the element to be written to the buffer and no blocking to occur.


> Kinesis Data Streams sink is not limiting throughput to the destination and therefore exceeding rate limits
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-25793
>                 URL: https://issues.apache.org/jira/browse/FLINK-25793
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Kinesis
>            Reporter: Zichen Liu
>            Assignee: Ahmed Hamdy
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.15.0
>
>
> h2. Bug:
> Kinesis Data Streams sink is not limiting throughput to the destination and therefore exceeding rate limits
> *Cause:*
> We are not throttling our requests downstream at all.
> We should monitor for requests that have failed with ThroughputExceeded exceptions and reduce the throughput accordingly.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)