[jira] [Commented] (FLINK-31135) ConfigMap DataSize went > 1 MB and cluster stopped working

["SwathiChandrashekar (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-31135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698321#comment-17698321 ] 

SwathiChandrashekar commented on FLINK-31135:
---------------------------------------------

[~sriramgr] , which is the configmap which failed in this scenario ?

If it is a pod-template configmap, then it indirectly depends on the user applied CR as the user defines entries of this file ( most of them ).

If it is a job specific config map ( which has the meta info of all the checkpoints pointers ) , if the retained checkpoints are very high, then I believe, we can hit this issue. 

> ConfigMap DataSize went > 1 MB and cluster stopped working
> ----------------------------------------------------------
>
>                 Key: FLINK-31135
>                 URL: https://issues.apache.org/jira/browse/FLINK-31135
>             Project: Flink
>          Issue Type: Bug
>          Components: Kubernetes Operator
>    Affects Versions: kubernetes-operator-1.2.0
>            Reporter: Sriram Ganesh
>            Priority: Major
>
> I am Flink Operator to manage clusters. Flink version: 1.15.2. Flink jobs failed with the below error. It seems the config map size went beyond 1 MB (default size). 
> Since it is managed by the operator and config maps are not updated with any manual intervention, I suspect it could be an operator issue. 
>  
> {code:java}
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://<IP>/api/v1/namespaces/<NS>/configmaps/<job>-config-map. Message: ConfigMap "<job>-config-map" is invalid: []: Too long: must have at most 1048576 bytes. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=[], message=Too long: must have at most 1048576 bytes, reason=FieldValueTooLong, additionalProperties={})], group=null, kind=ConfigMap, name=<job>-config-map, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=ConfigMap "<job>-config-map" is invalid: []: Too long: must have at most 1048576 bytes, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
> at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:673) ~[flink-dist-1.15.2.jar:1.15.2]
> at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:612) ~[flink-dist-1.15.2.jar:1.15.2]
> at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:560) ~[flink-dist-1.15.2.jar:1.15.2]
> at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521) ~[flink-dist-1.15.2.jar:1.15.2]
> at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:347) ~[flink-dist-1.15.2.jar:1.15.2]
> at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:327) ~[flink-dist-1.15.2.jar:1.15.2]
> at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleUpdate(BaseOperation.java:781) ~[flink-dist-1.15.2.jar:1.15.2]
> at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:183) ~[flink-dist-1.15.2.jar:1.15.2]
> at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:188) ~[flink-dist-1.15.2.jar:1.15.2]
> at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:130) ~[flink-dist-1.15.2.jar:1.15.2]
> at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:41) ~[flink-dist-1.15.2.jar:1.15.2]
> at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$attemptCheckAndUpdateConfigMap$11(Fabric8FlinkKubeClient.java:325) ~[flink-dist-1.15.2.jar:1.15.2]
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
> ... 3 more {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)