You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "mxm (via GitHub)" <gi...@apache.org> on 2023/11/01 14:17:41 UTC
[PR] [FLINK-33429] Tolerate missing metrics in the stabilization phase [flink-kubernetes-operator]
mxm opened a new pull request, #699:
URL: https://github.com/apache/flink-kubernetes-operator/pull/699
The new code for the 1.7.0 release introduces metric collection during the stabilization phase to allow sampling the observed true processing rate. Metrics might not be fully initialized during that phase, as evident through the error metrics. The following error is thrown:
```
java.lang.RuntimeException: Could not find required metric
NUM_RECORDS_OUT_PER_SEC for 667f5d5aa757fb217b92c06f0f5d2bf2
```
To prevent these errors shadowing actual errors, we should detect and ignore this recoverable exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33429] Tolerate missing metrics in the stabilization phase [flink-kubernetes-operator]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm merged PR #699:
URL: https://github.com/apache/flink-kubernetes-operator/pull/699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33429] Tolerate missing metrics in the stabilization phase [flink-kubernetes-operator]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #699:
URL: https://github.com/apache/flink-kubernetes-operator/pull/699#discussion_r1379822121
##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/exceptions/RecoverableException.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.autoscaler.exceptions;
+
+/** An exception for a recoverable scenario. */
+public class RecoverableException extends RuntimeException {
Review Comment:
Fair. At one point I actually had `NotReadyException` :) Renamed to `NotReadyException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33429] Tolerate missing metrics in the stabilization phase [flink-kubernetes-operator]
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #699:
URL: https://github.com/apache/flink-kubernetes-operator/pull/699#discussion_r1379839116
##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingMetricCollector.java:
##########
@@ -343,8 +346,19 @@ private double computeLagGrowthRate(
return (currentLag - lastLag) / timeDiff;
}
+ private Map<JobVertexID, Map<String, FlinkMetric>> queryFilteredMetricNames(
+ Context ctx, JobTopology topology, boolean isStabilizing) {
+ try {
+ return queryFilteredMetricNames(ctx, topology);
+ } catch (MetricNotFoundException e) {
+ if (isStabilizing) {
+ throw new RecoverableException(e);
+ }
+ throw e;
+ }
+ }
+
/** Query the available metric names for each job vertex. */
- @SneakyThrows
protected Map<JobVertexID, Map<String, FlinkMetric>> queryFilteredMetricNames(
Context ctx, JobTopology topology) {
Review Comment:
thanks for the clarification, sounds make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33429] Tolerate missing metrics in the stabilization phase [flink-kubernetes-operator]
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #699:
URL: https://github.com/apache/flink-kubernetes-operator/pull/699#discussion_r1379496109
##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/exceptions/RecoverableException.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.autoscaler.exceptions;
+
+/** An exception for a recoverable scenario. */
+public class RecoverableException extends RuntimeException {
Review Comment:
How about rename it to `IgnorableException` or `NotReadyException`?
Recoverable generally means that it can be recovered and will be tried again later. `JobAutoScalerImpl#scale` doesn't retry it, and just doesn't report `onError`. And regardless of whether it is a RecoverableException this time, the next control loop will perform the same logic.
Based on it, I think the `RecoverableException` isn't very accurate here. WDYT?
##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingMetricCollector.java:
##########
@@ -343,8 +346,19 @@ private double computeLagGrowthRate(
return (currentLag - lastLag) / timeDiff;
}
+ private Map<JobVertexID, Map<String, FlinkMetric>> queryFilteredMetricNames(
+ Context ctx, JobTopology topology, boolean isStabilizing) {
+ try {
+ return queryFilteredMetricNames(ctx, topology);
+ } catch (MetricNotFoundException e) {
+ if (isStabilizing) {
+ throw new RecoverableException(e);
+ }
+ throw e;
+ }
+ }
+
/** Query the available metric names for each job vertex. */
- @SneakyThrows
protected Map<JobVertexID, Map<String, FlinkMetric>> queryFilteredMetricNames(
Context ctx, JobTopology topology) {
Review Comment:
nit: could these 2 `queryFilteredMetricNames` methods be inlined? The old `queryFilteredMetricNames` method is only called by new `queryFilteredMetricNames` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33429] Tolerate missing metrics in the stabilization phase [flink-kubernetes-operator]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #699:
URL: https://github.com/apache/flink-kubernetes-operator/pull/699#issuecomment-1790597214
> This should clean up the logs nicely :)
Yes. I was also noticing it in the error counter which kept going up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33429] Tolerate missing metrics in the stabilization phase [flink-kubernetes-operator]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #699:
URL: https://github.com/apache/flink-kubernetes-operator/pull/699#discussion_r1379821385
##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingMetricCollector.java:
##########
@@ -343,8 +346,19 @@ private double computeLagGrowthRate(
return (currentLag - lastLag) / timeDiff;
}
+ private Map<JobVertexID, Map<String, FlinkMetric>> queryFilteredMetricNames(
+ Context ctx, JobTopology topology, boolean isStabilizing) {
+ try {
+ return queryFilteredMetricNames(ctx, topology);
+ } catch (MetricNotFoundException e) {
+ if (isStabilizing) {
+ throw new RecoverableException(e);
+ }
+ throw e;
+ }
+ }
+
/** Query the available metric names for each job vertex. */
- @SneakyThrows
protected Map<JobVertexID, Map<String, FlinkMetric>> queryFilteredMetricNames(
Context ctx, JobTopology topology) {
Review Comment:
I did not inline because this method is overridden in tests, e.g. `TestingScalingCollector`. Inlining the newly-added method into the protected method would change the behavior. I wanted to make sure that the newly added method is never overridden.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org