You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "1996fanrui (via GitHub)" <gi...@apache.org> on 2024/03/12 11:07:10 UTC
[PR] [FLINK-34655] Autoscaler doesn't work for flink 1.15 [flink-kubernetes-operator]
1996fanrui opened a new pull request, #797:
URL: https://github.com/apache/flink-kubernetes-operator/pull/797
flink-ubernetes-operator is committed to supporting the latest 4 flink minor versions, and autoscaler is a part of flink-ubernetes-operator. Currently, the latest 4 flink minor versions are 1.15, 1.16, 1.17 and 1.18.
But autoscaler doesn't work for flink 1.15.
## Root cause:
- [FLINK-28310](https://issues.apache.org/jira/browse/FLINK-28310) added some properties in IOMetricsInfo in flink-1.16
- IOMetricsInfo is a part of JobDetailsInfo
- JobDetailsInfo is necessary for autoscaler [1]
- flink's RestClient doesn't allow miss any property during deserializing the json
That means that the RestClient after 1.15 cannot fetch JobDetailsInfo for 1.15 jobs.
## How to fix it properly?
Flink side support ignore unknown properties.
[FLINK-33268](https://issues.apache.org/jira/browse/FLINK-33268) already do it. But I try run autoscaler with flink-1.15 job, it still doesn't work. Because the IOMetricsInfo added some properties, they are primitive type.
It should disable DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES as well. (Not sure whether it should be a seperate FLIP or it can be a part of FLIP-401 [2].)
## How to fix it in the short term?
1. Copy the latest RestMapperUtils and RestClient from master branch (It includes [FLINK-33268](https://issues.apache.org/jira/browse/FLINK-33268)) to flink-autoscaler module. (The copied class will be loaded first)
2. Disable DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES in RestMapperUtils#flexibleObjectMapper in copied class.
Based on these 2 steps, flink-1.15 works well with autoscaler. (I try it locally).
After DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES in RestMapperUtils#flexibleObjectMapper is disabled, and the corresponding code is released in flink side. flink-ubernetes-operator can remove these 2 copied classes.
[1] https://github.com/apache/flink-kubernetes-operator/blob/ede1a610b3375d31a2e82287eec67ace70c4c8df/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingMetricCollector.java#L109
[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-401%3A+REST+API+JSON+response+deserialization+unknown+field+tolerance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34655] Copy IOMetricsInfo to flink-autoscaler-standalone module [flink-kubernetes-operator]
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui merged PR #797:
URL: https://github.com/apache/flink-kubernetes-operator/pull/797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34655] Autoscaler doesn't work for flink 1.15 [flink-kubernetes-operator]
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on PR #797:
URL: https://github.com/apache/flink-kubernetes-operator/pull/797#issuecomment-1991512469
Instead of a new rest client you could also do what is done in the Kubernetes-operator module, see:
https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/runtime/rest/messages/job/metrics/IOMetricsInfo.java
By shadowing the class under the same package this will hide the incompatible implementation coming from flink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34655] Autoscaler doesn't work for flink 1.15 [flink-kubernetes-operator]
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on PR #797:
URL: https://github.com/apache/flink-kubernetes-operator/pull/797#issuecomment-1991513679
But still it will not make it work fully for 1.15 as the aggregated metrics need to be back ported on the Flink side (similar to other features required for the autoscaler)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34655] Autoscaler doesn't work for flink 1.15 [flink-kubernetes-operator]
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #797:
URL: https://github.com/apache/flink-kubernetes-operator/pull/797#issuecomment-1991537856
Thanks @gyfora for the valuable feedback!
> Instead of a new rest client you could also do what is done in the Kubernetes-operator module, see: https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/runtime/rest/messages/job/metrics/IOMetricsInfo.java
>
> By shadowing the class under the same package this will hide the incompatible implementation coming from flink
If so, autoscaler works with 1.15 when users uses the `flink-kubernetes-operator`, right?
I'm using the autoscaler Standalone, and I found 1.15 job doesn't work. In the short term, could I move the `IOMetricsInfo` to autoscaler module? IIUC, it will let both of `flink-kubernetes-operator` and `autoscaler Standalone` work.
In the long term, should we maintain the `IOMetricsInfo` class in the `flink-kubernetes-operator` repo or we disable `DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES` in the flink repo?
Looking forward to your opinion, and I'm happy to fix them.
> But still it will not make it work fully for 1.15 as the aggregated metrics need to be back ported on the Flink side (similar to other features required for the autoscaler)
Thanks for the reminder, we have a lot of jobs run with 1.15 version, so we are backing port them. Of course, we (our internal platform) only analyse them and don't scale them when the version before 1.18.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34655] Autoscaler doesn't work for flink 1.15 [flink-kubernetes-operator]
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on PR #797:
URL: https://github.com/apache/flink-kubernetes-operator/pull/797#issuecomment-1991544918
> Thanks @gyfora for the valuable feedback!
>
> > Instead of a new rest client you could also do what is done in the Kubernetes-operator module, see: https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/runtime/rest/messages/job/metrics/IOMetricsInfo.java
> > By shadowing the class under the same package this will hide the incompatible implementation coming from flink
>
> If so, autoscaler works with 1.15 when users uses the `flink-kubernetes-operator`, right?
>
> I'm using the autoscaler Standalone, and I found 1.15 job doesn't work. In the short term, could I move the `IOMetricsInfo` to autoscaler module? IIUC, it will let both of `flink-kubernetes-operator` and `autoscaler Standalone` work.
>
> In the long term, should we maintain the `IOMetricsInfo` class in the `flink-kubernetes-operator` repo or we disable `DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES` in the flink repo?
>
> Looking forward to your opinion, and I'm happy to fix them.
>
> > But still it will not make it work fully for 1.15 as the aggregated metrics need to be back ported on the Flink side (similar to other features required for the autoscaler)
>
> Thanks for the reminder, we have a lot of jobs run with 1.15 version, so we are backing port them. Of course, we (our internal platform) only analyse them and don't scale them when the version before 1.18.
Not sure if we need to have the IOMetricsInfo override in both packages to be safe actually. I think it has to be copied.
I can personally confirm that given the aggregated metrics back ported to 1.15 (easy to do) + the IOMetricsInfo, the autoscaler works with 1.15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org