You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "spuru9 (via GitHub)" <gi...@apache.org> on 2026/08/17 18:23:08 UTC
[PR] [FLINK-40401] Protect against large autoscaler config maps [flink-kubernetes-operator]
spuru9 opened a new pull request, #1181:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1181
## What is the purpose of the change
Protect against large autoscaler config maps. Improperly packaged autoscaler config maps can cause the Flink Kubernetes operator to fail with an OutOfMemoryException. We should put a size limit.
## Brief change log
- Added a bounded-read helper for decompressing autoscaler state (scaling history, scaling tracking, collected metrics) read from the `autoscaler-<app>` ConfigMap, capped at 20MB to match the existing YAML parser's own size limit
- Oversized or corrupted compressed values are now discarded through the existing fallback/migration path instead of being fully decompressed into memory
## Verifying this change
This change added tests and can be verified as follows:
- Added `KubernetesAutoScalerStateStoreTest#testDiscardOversizedCompressedHistory`, which injects an oversized compressed ConfigMap entry directly and verifies it is discarded rather than causing unbounded memory growth
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changes to the `CustomResourceDescriptors`: no
- Core observer or reconciler logic that is regularly executed: no (only the autoscaler state store's read path)
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Code)
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40401] Protect against large autoscaler config maps [flink-kubernetes-operator]
Posted by "rmetzger (via GitHub)" <gi...@apache.org>.
rmetzger merged PR #1181:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1181
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40401] Protect against large autoscaler config maps [flink-kubernetes-operator]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on code in PR #1181:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1181#discussion_r3802896825
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/autoscaler/state/KubernetesAutoScalerStateStore.java:
##########
@@ -80,6 +80,12 @@ public class KubernetesAutoScalerStateStore
@VisibleForTesting protected static final int MAX_CM_BYTES = 1000000;
+ /* Caps the size of a single decompressed value so that a crafted/corrupted ConfigMap
+ * entry (e.g. a gzip bomb) cannot exhaust operator memory during decompression. Matches
+ * the YAML loader's code point limit below, since larger content would be rejected by
+ * the YAML parser anyway. */
+ @VisibleForTesting protected static final int MAX_DECOMPRESSED_BYTES = 20 * 1024 * 1024;
Review Comment:
As ~4MB is real ceiling. Dropping the value to 8MB with some headspace. Let me know if thats ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40401] Protect against large autoscaler config maps [flink-kubernetes-operator]
Posted by "rmetzger (via GitHub)" <gi...@apache.org>.
rmetzger commented on PR #1181:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1181#issuecomment-5324241238
Is the maven in CI configured with some retries for downloading dependencies? it seems CI is pretty unstable, retrying should improve the situation?
We could also look into adding an additional maven mirror?
I don't know if this mirror is still maintained? https://maven-central.storage.googleapis.com/README.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40401] Protect against large autoscaler config maps [flink-kubernetes-operator]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on PR #1181:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1181#issuecomment-5318992118
Both the fails look unrelated. Might need a retrigger.
Also seeing these time of maven timeouts in FKO more often than flink is there a way to resolve these flakinesss?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40401] Protect against large autoscaler config maps [flink-kubernetes-operator]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on PR #1181:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1181#issuecomment-5324500615
Created https://issues.apache.org/jira/browse/FLINK-40406 for tracking. Can try retry first. Will pick after these PRs are resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40401] Protect against large autoscaler config maps [flink-kubernetes-operator]
Posted by "Dennis-Mircea (via GitHub)" <gi...@apache.org>.
Dennis-Mircea commented on code in PR #1181:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1181#discussion_r3802440251
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/autoscaler/state/KubernetesAutoScalerStateStore.java:
##########
@@ -388,15 +394,31 @@ private static String decompress(String compressed) {
try {
byte[] bytes = Base64.getDecoder().decode(compressed);
try (var zi = new GZIPInputStream(new ByteArrayInputStream(bytes))) {
- return IOUtils.toString(zi, StandardCharsets.UTF_8);
+ return readBounded(zi, MAX_DECOMPRESSED_BYTES);
}
} catch (Exception e) {
- LOG.warn("Error while decompressing scaling data, treating as uncompressed");
+ LOG.warn("Error while decompressing scaling data, treating as uncompressed", e);
Review Comment:
This log also fires on the normal migration path. Values stored before compression are not valid gzip, so `GZIPInputStream` throws and the code correctly falls back to treating them as uncompressed. Logging a full stack trace at WARN for that expected case is noise. Keep it without the exception, or attach it only when the value was expected to be compressed.
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/autoscaler/state/KubernetesAutoScalerStateStore.java:
##########
@@ -80,6 +80,12 @@ public class KubernetesAutoScalerStateStore
@VisibleForTesting protected static final int MAX_CM_BYTES = 1000000;
+ /* Caps the size of a single decompressed value so that a crafted/corrupted ConfigMap
+ * entry (e.g. a gzip bomb) cannot exhaust operator memory during decompression. Matches
+ * the YAML loader's code point limit below, since larger content would be rejected by
+ * the YAML parser anyway. */
+ @VisibleForTesting protected static final int MAX_DECOMPRESSED_BYTES = 20 * 1024 * 1024;
Review Comment:
20MB matches the YAML code point limit, not the heap. A legitimate value decompresses to at most ~4MB, because the compressed form is capped near 1MB (the ConfigMap limit plus the `MAX_CM_BYTES` trim) and the operator's own YAML compresses only ~2.5x to 4x (measured on real metric history), whereas a crafted or hijacked payload reaches its theoretical ~1032:1 maximum, which is the whole reason for a cap.
The default operator sets no heap limit, the peak per decode is ~32MB (ByteArrayOutputStream overshoot), and the cap bounds one value rather than the total, so concurrent poisoned CMs across the reconcile pool can still OOM under it. Deriving it from `Runtime.maxMemory()`, or dropping it near the ~4MB real ceiling, fits better than the parser limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40401] Protect against large autoscaler config maps [flink-kubernetes-operator]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on PR #1181:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1181#issuecomment-5326582654
@Dennis-Mircea Have addressed your comments. Can you recheck.
Also can someone retrigger the CI. The last one passed and this change should not be blocking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org