You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "HundalTaran (via GitHub)" <gi...@apache.org> on 2026/02/18 04:06:38 UTC
[PR] [FLINK-38096] AsyncSinkWriter may hang when configured with a custom rate limiting strategy [flink]
HundalTaran opened a new pull request, #27631:
URL: https://github.com/apache/flink/pull/27631
## What is the purpose of the change
In Async Writer, records are getting stuck on executing rate limiting. So, need to make sure that write should wait only if records are present in flight.
## Brief change log
Changed in method of AysncWriter.java class from mailboxExecutor.yield(); to yieldIfThereExistsInFlightRequests();
Also. added junit to cehck this yielding.
An example is provided with bucket4j to test rate limiting scenario.
## Verifying this change
Existing junits are getting passed.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / no)- no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - no
- The serializers: (yes / no / don't know) - no
- The runtime per-record code paths (performance sensitive): (yes / no / don't know) - no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - no
- The S3 file system connector: (yes / no / don't know) - no
## Documentation
- Does this pull request introduce a new feature? (yes / no) -no
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)- not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38096] AsyncSinkWriter may hang when configured with a custom rate limiting strategy [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #27631:
URL: https://github.com/apache/flink/pull/27631#issuecomment-3918476463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2458425d554ff0d26cbe5eca48f53add5fcb8f24",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2458425d554ff0d26cbe5eca48f53add5fcb8f24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2458425d554ff0d26cbe5eca48f53add5fcb8f24 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38096] AsyncSinkWriter may hang when configured with a custom rate limiting strategy [flink]
Posted by "HundalTaran (via GitHub)" <gi...@apache.org>.
HundalTaran commented on PR #27631:
URL: https://github.com/apache/flink/pull/27631#issuecomment-4010053580
I have checked the logs for build failure, but this failure does not seem due to code change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38096] AsyncSinkWriter may hang when configured with a custom rate limiting strategy [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on PR #27631:
URL: https://github.com/apache/flink/pull/27631#issuecomment-3921113046
Unapproved licenses
```
flink-examples/base-connector-examples/src/main/java/org/apache/flink/TokenBucketRateLimitingStrategy.java
flink-examples/base-connector-examples/src/main/java/org/apache/flink/TokenBucketProvider.java
flink-examples/base-connector-examples/src/main/java/org/apache/flink/AsyncSinkHangDemo.java
flink-examples/base-connector-examples/src/main/java/org/apache/flink/DummyAsyncSink.java
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38096] AsyncSinkWriter may hang when configured with a custom rate limiting strategy [flink]
Posted by "HundalTaran (via GitHub)" <gi...@apache.org>.
HundalTaran commented on code in PR #27631:
URL: https://github.com/apache/flink/pull/27631#discussion_r2844885079
##########
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java:
##########
@@ -353,7 +353,7 @@ private BasicRequestInfo createRequestInfo() {
private void flush() throws InterruptedException {
RequestInfo requestInfo = createRequestInfo();
while (rateLimitingStrategy.shouldBlock(requestInfo)) {
- mailboxExecutor.yield();
Review Comment:
It is explicitly mentioned in the Javadoc that until or unless another command is not triggered :
/**
* This method starts running the command at the head of the mailbox and is intended to be used
* by the mailbox thread to yield from a currently ongoing action to another command. The method
* blocks until another command to run is available in the mailbox and must only be called from
* the mailbox thread. Must only be called from the mailbox thread to not violate the
* single-threaded execution model.
*
* @throws InterruptedException on interruption.
* @throws IllegalStateException if the mailbox is closed and can no longer supply runnables for
* yielding.
* @throws FlinkRuntimeException if executed {@link RunnableWithException} thrown an exception.
*/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38096] AsyncSinkWriter may hang when configured with a custom rate limiting strategy [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27631:
URL: https://github.com/apache/flink/pull/27631#discussion_r2840979010
##########
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java:
##########
@@ -353,7 +353,7 @@ private BasicRequestInfo createRequestInfo() {
private void flush() throws InterruptedException {
RequestInfo requestInfo = createRequestInfo();
while (rateLimitingStrategy.shouldBlock(requestInfo)) {
- mailboxExecutor.yield();
Review Comment:
I am curious, is the yield behaviour an explicit contract documented on the yield method or due to the implementations of yield?
If it is not explicit, we should update the javadoc for the yield method in the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38096] AsyncSinkWriter may hang when configured with a custom rate limiting strategy [flink]
Posted by "HundalTaran (via GitHub)" <gi...@apache.org>.
HundalTaran commented on PR #27631:
URL: https://github.com/apache/flink/pull/27631#issuecomment-3942239621
> Unapproved licenses
>
> ```
> flink-examples/base-connector-examples/src/main/java/org/apache/flink/TokenBucketRateLimitingStrategy.java
> flink-examples/base-connector-examples/src/main/java/org/apache/flink/TokenBucketProvider.java
> flink-examples/base-connector-examples/src/main/java/org/apache/flink/AsyncSinkHangDemo.java
> flink-examples/base-connector-examples/src/main/java/org/apache/flink/DummyAsyncSink.java
> ```
Thanks for the comments, will work on that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38096] AsyncSinkWriter may hang when configured with a custom rate limiting strategy [flink]
Posted by "HundalTaran (via GitHub)" <gi...@apache.org>.
HundalTaran closed pull request #27631: [FLINK-38096] AsyncSinkWriter may hang when configured with a custom rate limiting strategy
URL: https://github.com/apache/flink/pull/27631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org