You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "mxm (via GitHub)" <gi...@apache.org> on 2024/02/28 12:33:17 UTC
[PR] [FLINK-34152] Add an option to scale memory when downscaling [flink-kubernetes-operator]
mxm opened a new pull request, #786:
URL: https://github.com/apache/flink-kubernetes-operator/pull/786
This adds an option to increase heap and managed memory when removing TaskManagers. The scaling is applied after adjusting the memory pools (heap, metaspace, network, managed) and only affects heap and managed memory.
The reason for adding this functionality is that the likelihood of running into memory constrained scenarios when downscaling is increased after applying memory tuning. As a precaution, we temporarily increase the total cluster
memory proportionally to the removed TaskManagers up to the maximum allowed TaskManager memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34152] Add an option to scale memory when downscaling [flink-kubernetes-operator]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm merged PR #786:
URL: https://github.com/apache/flink-kubernetes-operator/pull/786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34152] Add an option to scale memory when downscaling [flink-kubernetes-operator]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #786:
URL: https://github.com/apache/flink-kubernetes-operator/pull/786#discussion_r1511334711
##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/tuning/MemoryTuning.java:
##########
@@ -148,8 +149,11 @@ public static ConfigChanges tuneTaskManagerHeapMemory(
final long flinkMemoryDiffBytes = heapDiffBytes + managedDiffBytes + networkDiffBytes;
// Update total memory according to memory diffs
- final MemorySize totalMemory =
+ MemorySize totalMemory =
new MemorySize(maxMemoryBySpec.getBytes() - memBudget.getRemaining());
+ totalMemory =
+ MemoryScaling.applyMemoryScaling(
+ totalMemory, maxMemoryBySpec, context, scalingSummaries, evaluatedMetrics);
Review Comment:
Good catch, the scaling is heap only. Managed memory is never adjusted, unless none is used at all (then it is set to zero) or the option to maximize managed memory is turned on. I've adjusted the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34152] Add an option to scale memory when downscaling [flink-kubernetes-operator]
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #786:
URL: https://github.com/apache/flink-kubernetes-operator/pull/786#discussion_r1511285884
##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/tuning/MemoryTuning.java:
##########
@@ -148,8 +149,11 @@ public static ConfigChanges tuneTaskManagerHeapMemory(
final long flinkMemoryDiffBytes = heapDiffBytes + managedDiffBytes + networkDiffBytes;
// Update total memory according to memory diffs
- final MemorySize totalMemory =
+ MemorySize totalMemory =
new MemorySize(maxMemoryBySpec.getBytes() - memBudget.getRemaining());
+ totalMemory =
+ MemoryScaling.applyMemoryScaling(
+ totalMemory, maxMemoryBySpec, context, scalingSummaries, evaluatedMetrics);
Review Comment:
The memory scaling config specifies that both managed and heap memory is scaled based on the TM scaling factor. But it seems like we only scale the heap as the managed memory size is computed before this.
Am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34152] Add an option to scale memory when downscaling [flink-kubernetes-operator]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #786:
URL: https://github.com/apache/flink-kubernetes-operator/pull/786#issuecomment-1983822516
@1996fanrui I'm merging but I'll take any comments from your side that you might have reviewing this later on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34152] Add an option to scale memory when downscaling [flink-kubernetes-operator]
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #786:
URL: https://github.com/apache/flink-kubernetes-operator/pull/786#issuecomment-1987512556
> @1996fanrui I'm merging but I'll take any comments from your side that you might have reviewing this later on.
Thanks @mxm for the improvement and ping! Sorry for the late response here. I just finished my vacation, I will take a look these 2 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org