You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "becketqin (via GitHub)" <gi...@apache.org> on 2025/02/24 18:02:29 UTC
[PR] [FLINK-36585][connector/common] SplitFetcherManager.close() should not chain runnables to the element queue future in a tight loop [flink]
becketqin opened a new pull request, #26205:
URL: https://github.com/apache/flink/pull/26205
## What is the purpose of the change
This is a re-commit of the patch ( #25569 ) after fixing the accidentally changed timeout unit of the SplitFetcherManager close timeout.
## Brief change log
Changes in the Original Fix:
This patch fixes an issue introduced in https://github.com/apache/flink/pull/25130. In SplitFetcherManager.close(), the element queue draining thread was chaining the runnables to the element queue availability future in a tight loop. This causes problem (e.g. OOM, high CPU util) when the fetcher threads do not shutdown quickly.
This patch changes the tight async loop to a blocking loop.
Changes in the followup patch fixes the accidentally changed timeout unit of the SplitFetcherManager close timeout.
## Verifying this change
Unit tests have been added.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (No)
- If yes, how is the feature documented? (No)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36585][connector/common] SplitFetcherManager.close() should not chain runnables to the element queue future in a tight loop [flink]
Posted by "becketqin (via GitHub)" <gi...@apache.org>.
becketqin merged PR #26205:
URL: https://github.com/apache/flink/pull/26205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36585][connector/common] SplitFetcherManager.close() should not chain runnables to the element queue future in a tight loop [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26205:
URL: https://github.com/apache/flink/pull/26205#discussion_r1971849410
##########
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/fetcher/SplitFetcherManager.java:
##########
@@ -269,19 +271,32 @@ public synchronized void close(long timeoutMs) throws Exception {
// fetcher threads blocking on putting batches into the element queue.
executors.submit(
() -> {
- while (fetchersToShutDown.get() > 0
- && System.currentTimeMillis() - startTime < timeoutMs) {
- elementsQueue
- .getAvailabilityFuture()
- .thenRun(() -> elementsQueue.poll().recycle());
+ long timeElapsed = System.currentTimeMillis() - startTime;
+ while (fetchersToShutDown.get() > 0 && timeElapsed < timeoutMs) {
+ try {
+ elementsQueue
+ .getAvailabilityFuture()
+ .thenRun(() -> elementsQueue.poll().recycle())
+ .get(timeoutMs - timeElapsed, TimeUnit.MILLISECONDS);
+ } catch (ExecutionException ee) {
+ // Ignore the exception and continue.
+ } catch (Exception e) {
+ LOG.warn(
+ "Received exception when waiting for the fetchers to "
+ + "shutdown.",
+ e);
+ break;
+ }
+ timeElapsed = System.currentTimeMillis() - startTime;
Review Comment:
Could we put out a log if we have timed out with elements still on the queue, or would the await termination `LOG.warn(` catch this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36585][connector/common] SplitFetcherManager.close() should not chain runnables to the element queue future in a tight loop [flink]
Posted by "becketqin (via GitHub)" <gi...@apache.org>.
becketqin commented on PR #26205:
URL: https://github.com/apache/flink/pull/26205#issuecomment-2680018404
@PatrickRen Can you take another look? cc @snuyanzin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36585][connector/common] SplitFetcherManager.close() should not chain runnables to the element queue future in a tight loop [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26205:
URL: https://github.com/apache/flink/pull/26205#issuecomment-2679306684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c1746f6f4df52e9397586244e4e7c6cc7d2d3e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0c1746f6f4df52e9397586244e4e7c6cc7d2d3e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c1746f6f4df52e9397586244e4e7c6cc7d2d3e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36585][connector/common] SplitFetcherManager.close() should not chain runnables to the element queue future in a tight loop [flink]
Posted by "becketqin (via GitHub)" <gi...@apache.org>.
becketqin commented on code in PR #26205:
URL: https://github.com/apache/flink/pull/26205#discussion_r1972562795
##########
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/fetcher/SplitFetcherManager.java:
##########
@@ -269,19 +271,32 @@ public synchronized void close(long timeoutMs) throws Exception {
// fetcher threads blocking on putting batches into the element queue.
executors.submit(
() -> {
- while (fetchersToShutDown.get() > 0
- && System.currentTimeMillis() - startTime < timeoutMs) {
- elementsQueue
- .getAvailabilityFuture()
- .thenRun(() -> elementsQueue.poll().recycle());
+ long timeElapsed = System.currentTimeMillis() - startTime;
+ while (fetchersToShutDown.get() > 0 && timeElapsed < timeoutMs) {
+ try {
+ elementsQueue
+ .getAvailabilityFuture()
+ .thenRun(() -> elementsQueue.poll().recycle())
+ .get(timeoutMs - timeElapsed, TimeUnit.MILLISECONDS);
+ } catch (ExecutionException ee) {
+ // Ignore the exception and continue.
+ } catch (Exception e) {
+ LOG.warn(
+ "Received exception when waiting for the fetchers to "
+ + "shutdown.",
+ e);
+ break;
+ }
+ timeElapsed = System.currentTimeMillis() - startTime;
Review Comment:
Exactly, the await termination `LOG.warn()` would catch it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org