You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "raulpardo (via GitHub)" <gi...@apache.org> on 2025/02/26 16:17:22 UTC
[PR] [hotfix] fix data race in method in [flink]
raulpardo opened a new pull request, #26219:
URL: https://github.com/apache/flink/pull/26219
## What is the purpose of the change
This is a hotfix fixing a data race in `drain()` method in `TaskMailboxImpl.java`. This can lead to undesired executions. For instance, consider the that mailbox thread is adding a `mail` by executing `putFirst`, and concurrently another thread is executing `drain()`. This can produce the following execution:
1. The read of `batch` [here](https://github.com/apache/flink/blob/34a7734c489b080d34ff2194a29d3c1d25d3ab45/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java#L246) is executed before `addFirst(mail)` ([in this line](https://github.com/apache/flink/blob/34a7734c489b080d34ff2194a29d3c1d25d3ab45/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java#L210C13-L210C18)) adds the element,
2. Then, `mail` is added to `batch` (consequentely, `mail` is not in `drainedMails`)
3. Finally, `batch.clear()` is executed.
This execution results in missing `mail` (as it is neither included in `batch` nor `drainedMails`).
This hotfix PR resolves the data race by including the read to `batch` and its clearing within the critical section in the `drain()` method.
## Brief change log
- Reading and clearing variable `batch` in the method `drain()` in `TaskMailboxImpl.java` are moved into the critical section within the method.
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: don't know
- The runtime per-record code paths (performance sensitive): don't know
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: don't know
- The S3 file system connector: don't know
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java` [flink]
Posted by "raulpardo (via GitHub)" <gi...@apache.org>.
raulpardo commented on code in PR #26219:
URL: https://github.com/apache/flink/pull/26219#discussion_r1976574603
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -243,11 +243,11 @@ private Mail takeOrNull(Deque<Mail> queue, int priority) {
@Override
public List<Mail> drain() {
- List<Mail> drainedMails = new ArrayList<>(batch);
- batch.clear();
Review Comment:
This is a good idea that would solve all data races. However, I noticed that the current implementation seems to be acquiring locks depending on what thread is executing the method by invoking `checkIsMailboxThread()`. It might be that this was introduced for performance reasons; as acquiring the lock is a rather slow operation. But this is not entirely clear to me, as there are other methods like `createBatch` that acquire the lock even though the first instruction is to check is to ensure that only the mailbox thread is executing this.
This is why I reported only the data race on the `drain` method, as there is no instruction indicating that only the mailbox thread can execute that method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java` [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26219:
URL: https://github.com/apache/flink/pull/26219#issuecomment-2685603442
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9400657bdbc244a1eb03d157b13de9ca5f7f7555",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9400657bdbc244a1eb03d157b13de9ca5f7f7555",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9400657bdbc244a1eb03d157b13de9ca5f7f7555 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java` [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26219:
URL: https://github.com/apache/flink/pull/26219#issuecomment-2926651059
This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.
If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/
If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java` [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26219:
URL: https://github.com/apache/flink/pull/26219#issuecomment-4278535783
This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.
If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/
If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java` [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #26219: [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java`
URL: https://github.com/apache/flink/pull/26219
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java` [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26219:
URL: https://github.com/apache/flink/pull/26219#discussion_r1975168566
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -243,11 +243,11 @@ private Mail takeOrNull(Deque<Mail> queue, int priority) {
@Override
public List<Mail> drain() {
- List<Mail> drainedMails = new ArrayList<>(batch);
- batch.clear();
Review Comment:
should we be eagerly grabbing the lock in all these methods so all processing is done under the lock? for example all references to batch should be under the lock as the value could change under us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java` [flink]
Posted by "raulpardo (via GitHub)" <gi...@apache.org>.
raulpardo commented on PR #26219:
URL: https://github.com/apache/flink/pull/26219#issuecomment-2692658830
> can we have a unit test?
Sure, I can write a test. I am looking at [this page](https://github.com/apache/flink/tree/master/flink-end-to-end-tests#adding-a-test-case) to read about how to write unit tests for flink. I would like to add a java test, but there is no information on how to run Java tests. Is there some documentation on this? (if it is possible to write Junit5 test it would be better and the test must be run repeatedly for the bug to appear)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java` [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26219:
URL: https://github.com/apache/flink/pull/26219#issuecomment-3435364502
This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.
If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/
If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java` [flink]
Posted by "raulpardo (via GitHub)" <gi...@apache.org>.
raulpardo commented on PR #26219:
URL: https://github.com/apache/flink/pull/26219#issuecomment-2685614739
> ## CI report:
>
> * [9400657](https://github.com/apache/flink/commit/9400657bdbc244a1eb03d157b13de9ca5f7f7555) UNKNOWN
>
>
> Bot commands
The link to the commit is unkown because I amended the commit message to clarify the subject of the PR. The new commit link is https://github.com/apache/flink/commit/7e3700a061b28be16b1661b4573b24400461ab2d, and the changes are the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java` [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26219:
URL: https://github.com/apache/flink/pull/26219#discussion_r1975168566
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -243,11 +243,11 @@ private Mail takeOrNull(Deque<Mail> queue, int priority) {
@Override
public List<Mail> drain() {
- List<Mail> drainedMails = new ArrayList<>(batch);
- batch.clear();
Review Comment:
should we be eagerly grabbing the lock in all these methods so all processing is done under the lock? for example all references to batch should be under the lock as the value could change under us. I was thinking we could synchronize all the methods referencing batch, so we do not need the lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] fix data race in method `drain()` in `TaskMailboxImpl.java` [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26219:
URL: https://github.com/apache/flink/pull/26219#issuecomment-4495731552
This PR has been closed since it has not had any activity in 120 days.
If you feel like this was a mistake, or you would like to continue working on it,
please feel free to re-open the PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org