You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "BeiWeiDeDengDai (via GitHub)" <gi...@apache.org> on 2023/11/23 02:24:24 UTC
[I] [Feature] Flink consumption backlog is quite serious, but jobManager will continue to scan paymon snapshots, ultimately resulting in jobManager OOM [incubator-paimon]
BeiWeiDeDengDai opened a new issue, #2370:
URL: https://github.com/apache/incubator-paimon/issues/2370
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
I logged and observed the consumption situation on ContinuousFileSplitEnumerator
The first batch of data 15:38:29:5600w shards: 600 shards to be allocated 580
The second batch of data 15:43:19:5500w Partitions: 600 Partitions to be allocated 1118
The third batch of data 15:49:29 5500w Number of slices: 600 to be allocated 1652
### Solution
The reason is that there is backpressure in downstream processing. We hope to optimize the jobManager based on downstream processing speed and not add new shards ,Or there could be a better solution
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Flink consumption backlog is quite serious, but jobManager will continue to scan paymon snapshots, ultimately resulting in jobManager OOM [incubator-paimon]
Posted by "BeiWeiDeDengDai (via GitHub)" <gi...@apache.org>.
BeiWeiDeDengDai commented on issue #2370:
URL: https://github.com/apache/incubator-paimon/issues/2370#issuecomment-1823750226
<img width="954" alt="image" src="https://github.com/apache/incubator-paimon/assets/35143428/c1ca5b32-8aed-4c7c-92c2-9a1de53ffc5a">
<img width="959" alt="image" src="https://github.com/apache/incubator-paimon/assets/35143428/a555330b-e928-4d2b-bbf5-e2a707170ae6">
<img width="962" alt="image" src="https://github.com/apache/incubator-paimon/assets/35143428/17f43f6f-c7fa-4860-ae85-2665473ae039">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Flink consumption backlog is quite serious, but jobManager will continue to scan paymon snapshots, ultimately resulting in jobManager OOM [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #2370: [Feature] Flink consumption backlog is quite serious, but jobManager will continue to scan paymon snapshots, ultimately resulting in jobManager OOM
URL: https://github.com/apache/incubator-paimon/issues/2370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org