You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/11/23 07:34:45 UTC
[GitHub] [flink] HuangZhenQiu opened a new pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
HuangZhenQiu opened a new pull request #8952:
URL: https://github.com/apache/flink/pull/8952
## What is the purpose of the change
Add failure rate for resource manager, so that every type of Resource Managers can handle with maximum failed executers.
## Brief change log
- Add a failure rater to record failure numbers in last one minute.
- Update Resource Manager to use maximum failure rate threshold to limit the resource allocation.
## Verifying this change
- Added unit test in Resource Manager to verify the functionality of maximum failure rate threshold
enforcement
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547825381
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -246,6 +269,10 @@ private void requestNewWorker(WorkerResourceSpec workerResourceSpec) {
workerResourceSpec,
count,
exception);
+ if (recordWorkerFailure()) {
+ // if exceed failure rate try to slow down
+ tryResetWorkerCreationCoolDown();
+ }
Review comment:
What is the concern on having different behaviors in `onWorkerTerminated`?
IIUC, this feature is a protection agains container start-up failures. The current implementation counts all the terminated containers as failed, including those that are terminated normally as expected.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-751315528
@xintongsong
Thanks for the comments. I do like the simplified version after grouping classes and utils together. For the TimestampBasedFailureRater, I would prefer to leave it as it is. Otherwise, we need to split the 100 lines of code into ThresholdMeter, TimestampBasedThresholdMeter, and FailureRater. I think we can keep it simple and refactor it once ThresholdMeter, TimestampBasedThresholdMeter can be usable in other components.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547768470
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/FailureRaterUtil.java
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.ResourceManagerOptions;
+
+/**
+ * Failure rate util.
+ */
+public class FailureRaterUtil {
+
+ public static FailureRater createFailureRater(
+ Configuration configuration) {
+ double rate = configuration.getDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE);
+ if (rate < 0) {
+ return new NoFailureRater(60);
+ } else {
+ int maximumRate = Double.valueOf(rate).intValue();
+ return new TimestampBasedFailureRater(maximumRate < rate ? maximumRate + 1 : maximumRate, Time.minutes(1));
+ }
Review comment:
Agree. After combing the failure rate as the threshold for slowing down requests. I think we should remove the NoFailureRater and make sure the value > 0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
}, {
"hash" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535",
"triggerID" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"triggerType" : "PUSH"
}, {
"hash" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11632",
"triggerID" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 50dd8154daca82c54bc3d0decc8e94b6b7cba8d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535)
* 8b920eb9c1f70c08f663154e15c60f6ed331394b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11632)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-752320806
On more thing, we need to update the PR w.r.t. the community's new formatting rules.
Please check this [thread](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-New-formatting-rules-are-now-in-effect-td47441.html).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r550417478
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -301,7 +353,25 @@ private boolean clearStateForWorker(ResourceID resourceId) {
return true;
}
- private void requestWorkerIfRequired() {
+ private void tryResetWorkerCreationCoolDown() {
+ if (workerCreationCoolDown.isDone()) {
+ log.info(
+ "Reaching max start worker failure rate. Will not retry creating worker in {}.",
+ workerCreationRetryInterval);
+ workerCreationCoolDown = new CompletableFuture<>();
+ getMainThreadExecutor()
+ .scheduleRunAsync(
+ () -> workerCreationCoolDown.complete(null),
+ workerCreationRetryInterval.getSize());
Review comment:
Yes, then what's the concern?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-705393891
@HuangZhenQiu
I think you can reopen it (at the bottom of the comment box to the left), once you start again working on it.
Let me me know if you do not have this button `Reopen and comment` and you start to work on, I will reopen the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 17290d66d3ad93815bdfea6343857b62ed182772 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r549938770
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
Review comment:
Yes, agree we don't need to have three layers. The latest PR contains only two layers, Meter and TimestampBasedFailureRater. Why I don't want to use ThresholdMeter as the implementation class name? This two method checkThreshold and fromConfiguration has container failure related error message and configuration key. It is not generic enough to put into ThresholdMeter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548939375
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/ResourceManagerOptions.java
##########
@@ -67,6 +69,33 @@
"for streaming workloads, which may fail if there are not enough slots. Note that this configuration option does not take " +
"effect for standalone clusters, where how many slots are allocated is not controlled by Flink.");
+ /**
+ * Defines the maximum number of worker (YARN / Mesos / Kubernetes) failures per minute before rejecting subsequent worker
+ * requests until the failure rate falls below the maximum. It is to quickly catch external dependency caused
+ * workers failure and wait for retry interval before sending new request. By default, the value is set to 10/min.
+ */
+ public static final ConfigOption<Double> MAXIMUM_WORKERS_FAILURE_RATE = ConfigOptions
+ .key("resourcemanager.start-worker.max-failure-rate")
+ .doubleType()
+ .defaultValue(10.0)
+ .withDescription("Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting" +
+ " subsequent worker requests until the failure rate falls below the maximum. It is to quickly catch" +
+ " external dependency caused workers failure and terminate job accordingly." +
+ " By default, the value is set to 10/min.");
+
+ /**
+ * Defines the worker creation interval in milliseconds. In case of worker creation failures, we should wait for an interval before
+ * trying to create new workers when the failure rate exceeds. Otherwise, ActiveResourceManager will always re-requesting
+ * the worker, which keeps the main thread busy.
+ */
+ public static final ConfigOption<Duration> WORKER_CREATION_RETRY_INTERVAL = ConfigOptions
+ .key("resourcemanager.start-worker.retry-interval")
+ .durationType()
+ .defaultValue(Duration.ofMillis(30))
Review comment:
Updated to 3 seconds as the default value of original Kubernetes interval.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547801045
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManagerTest.java
##########
@@ -378,15 +382,69 @@ public void testOnError() throws Exception {
}};
}
- private static class Context {
+ @Test
+ public void testWorkerCreationInterval() throws Exception {
+ Configuration configuration = new Configuration();
+ configuration.setDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE, 1);
+ configuration.setLong(ResourceManagerOptions.WORKER_CREATION_RETRY_INTERVAL,
+ WORKER_CREATION_INTERVAL.toMilliseconds());
+ new Context(configuration) {{
Review comment:
Agree. But these configs are used for the initialization of the resource manager which is before calling the runTest.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e2d99f428b8c04df9f7709f9454898e2429d3cf0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086)
* f5c053117296cab1417bfbf98a0a8e63f48def5b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547782447
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
##########
@@ -143,6 +150,8 @@
private final ResourceManagerMetricGroup resourceManagerMetricGroup;
+ protected final FailureRater failureRater;
Review comment:
Make sense.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
##########
@@ -845,6 +856,10 @@ private void registerTaskExecutorMetrics() {
resourceManagerMetricGroup.gauge(
MetricNames.NUM_REGISTERED_TASK_MANAGERS,
() -> (long) taskExecutors.size());
+
+ resourceManagerMetricGroup.meter(
Review comment:
Agree. Revised.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* d58927642909f50571ed6242605aac564e074f89 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
}, {
"hash" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535",
"triggerID" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"triggerType" : "PUSH"
}, {
"hash" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11632",
"triggerID" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"triggerType" : "PUSH"
}, {
"hash" : "27b18b35b2826fa5fc721c3f19264a8a6cb10369",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11920",
"triggerID" : "27b18b35b2826fa5fc721c3f19264a8a6cb10369",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 8b920eb9c1f70c08f663154e15c60f6ed331394b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11632)
* 27b18b35b2826fa5fc721c3f19264a8a6cb10369 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11920)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* adf63fa7c5c73eecf4ae0237cb3f0fde2d616082 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236)
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-737375866
@xintongsong Would you please help to review this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547734287
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/ResourceManagerOptions.java
##########
@@ -67,6 +67,33 @@
"for streaming workloads, which may fail if there are not enough slots. Note that this configuration option does not take " +
"effect for standalone clusters, where how many slots are allocated is not controlled by Flink.");
+ /**
+ * Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting subsequent worker
+ * requests until the failure rate falls below the maximum. It is to quickly catch external dependency caused
+ * workers failure and wait for retry interval before sending new request. Be default, -1.0 is set to disable the feature.
+ */
+ public static final ConfigOption<Double> MAXIMUM_WORKERS_FAILURE_RATE = ConfigOptions
+ .key("resourcemanager.maximum-workers-failure-rate")
+ .doubleType()
+ .defaultValue(-1.0)
+ .withDescription("Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting" +
+ " subsequent worker requests until the failure rate falls below the maximum. It is to quickly catch" +
+ " external dependency caused workers failure and terminate job accordingly." +
+ " Be default, -1.0 is set to disable the feature.");
+
+ /**
+ * Defines the worker creation interval in milliseconds. In case of worker creation failures, we should wait for an interval before
+ * trying to create new workers when the failure rate exceeds. Otherwise, ActiveResourceManager will always re-requesting
+ * the worker, which keeps the main thread busy.
+ */
+ public static final ConfigOption<Long> WORKER_CREATION_RETRY_INTERVAL = ConfigOptions
+ .key("resourcemanager.workers-creation-retry-interval")
+ .longType()
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548939312
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval) {
+ this(maximumFailureRate, failureInterval, System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval, Supplier<Long> customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(System.currentTimeMillis());
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() / (failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public void checkAgainstThreshold() throws ThresholdExceedException {
+ if (getCurrentFailureRate() >= maximumFailureRate) {
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547855489
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManagerTest.java
##########
@@ -378,15 +382,69 @@ public void testOnError() throws Exception {
}};
}
- private static class Context {
+ @Test
+ public void testWorkerCreationInterval() throws Exception {
+ Configuration configuration = new Configuration();
+ configuration.setDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE, 1);
+ configuration.setLong(ResourceManagerOptions.WORKER_CREATION_RETRY_INTERVAL,
+ WORKER_CREATION_INTERVAL.toMilliseconds());
+ new Context(configuration) {{
Review comment:
Sorry, what I mean is the initialization of the resource manager is before the testMethod.run(). We can't override the config in RunnableWithException to change the behavior of rm.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManagerTest.java
##########
@@ -378,15 +382,69 @@ public void testOnError() throws Exception {
}};
}
- private static class Context {
+ @Test
+ public void testWorkerCreationInterval() throws Exception {
+ Configuration configuration = new Configuration();
+ configuration.setDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE, 1);
+ configuration.setLong(ResourceManagerOptions.WORKER_CREATION_RETRY_INTERVAL,
+ WORKER_CREATION_INTERVAL.toMilliseconds());
+ new Context(configuration) {{
+ final AtomicInteger requestCount = new AtomicInteger(0);
+
+ final List<ResourceID> tmResourceIds = new ArrayList<>();
+ tmResourceIds.add(ResourceID.generate());
+ tmResourceIds.add(ResourceID.generate());
+
+ final List<CompletableFuture<Long>> requestWorkerFromDriverFutures = new ArrayList<>();
+ requestWorkerFromDriverFutures.add(new CompletableFuture<>());
+ requestWorkerFromDriverFutures.add(new CompletableFuture<>());
+
+ driverBuilder.setRequestResourceFunction(taskExecutorProcessSpec -> {
+ int idx = requestCount.getAndIncrement();
+ assertThat(idx, lessThan(2));
+
+ requestWorkerFromDriverFutures.get(idx).complete(System.currentTimeMillis());
+ return CompletableFuture.completedFuture(tmResourceIds.get(idx));
+ });
+
+ slotManagerBuilder.setGetRequiredResourcesSupplier(() -> Collections.singletonMap(WORKER_RESOURCE_SPEC, 1));
- final Configuration flinkConfig = new Configuration();
+ runTest(() -> {
+ // received worker request, verify requesting from driver
+ CompletableFuture<Boolean> startNewWorkerFuture = runInMainThread(() ->
+ getResourceManager().startNewWorker(WORKER_RESOURCE_SPEC));
+ long t1 = requestWorkerFromDriverFutures.get(0).get(TIMEOUT_SEC, TimeUnit.SECONDS);
+ assertThat(startNewWorkerFuture.get(TIMEOUT_SEC, TimeUnit.SECONDS), is(true));
+
+ // first worker failed before register, verify requesting another worker from driver
+ runInMainThread(() -> getResourceManager().onWorkerTerminated(tmResourceIds.get(0), "terminate for testing"));
Review comment:
Agree. Added another test case accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 17290d66d3ad93815bdfea6343857b62ed182772 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294)
* 67e9e1e4c013b78eaee9fc9d215ef400bba5cc59 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r549927391
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
Review comment:
I believe the three layers you mean are the following.
1. Meter (interface)
2. ThresholdMeter (interface)
3. TimestampBasedThresholdMeter (implementation)
My questions is, why do we need 2) and 3) separately? What if we consider `ThresholdMeter` an implementation of `Meter` and have the following two layers?
1. Meter (interface)
2. ThresholdMeter (implementation)
I'm proposing this because:
* There's no other implementation of `ThresholdMeter` except for `TimestampBasedThresholdMeter`.
* Looking into the implementation of `TimestampBasedFailureRater`, there's nothing specifically related to *Failure*, and shouldn't the concept *Meter* already *TimestampBased*?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong closed pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong closed pull request #8952:
URL: https://github.com/apache/flink/pull/8952
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* d58927642909f50571ed6242605aac564e074f89 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349)
* 70b43a088878601fa074245f88800b163641d6f0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547784241
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -259,6 +286,22 @@ private void requestNewWorker(WorkerResourceSpec workerResourceSpec) {
}));
}
+ private void tryResetWorkerCreationCoolDown() {
+ if (workerCreationCoolDown.isDone()) {
+ log.info("Worker creation failed. Will not retry creating worker in {}.", workerCreationRetryInterval);
Review comment:
Revised
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -259,6 +286,22 @@ private void requestNewWorker(WorkerResourceSpec workerResourceSpec) {
}));
}
+ private void tryResetWorkerCreationCoolDown() {
+ if (workerCreationCoolDown.isDone()) {
+ log.info("Worker creation failed. Will not retry creating worker in {}.", workerCreationRetryInterval);
+ workerCreationCoolDown = new CompletableFuture<>();
+ getMainThreadExecutor().schedule(
Review comment:
Revised.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547770123
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.ArrayDeque;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements FailureRater {
+ private static final int DEFAULT_TIMESTAMP_SIZE = 300;
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final int maximumFailureRate;
+ private final Time failureInterval;
+ private final ArrayDeque<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(int maximumFailureRate, Time failureInterval) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>(maximumFailureRate > 0 ? maximumFailureRate : DEFAULT_TIMESTAMP_SIZE);
+ }
+
+ @Override
+ public void markEvent() {
+ markEvent(System.currentTimeMillis());
Review comment:
Revised.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* f59351ab8f34947c26b357ef3f87e80a85996609 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262)
* 7cffa19fd89147eb546988f22f2f98c911e044b4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-507785614
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 27b18b35b2826fa5fc721c3f19264a8a6cb10369 (Fri May 28 07:06:17 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547855489
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManagerTest.java
##########
@@ -378,15 +382,69 @@ public void testOnError() throws Exception {
}};
}
- private static class Context {
+ @Test
+ public void testWorkerCreationInterval() throws Exception {
+ Configuration configuration = new Configuration();
+ configuration.setDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE, 1);
+ configuration.setLong(ResourceManagerOptions.WORKER_CREATION_RETRY_INTERVAL,
+ WORKER_CREATION_INTERVAL.toMilliseconds());
+ new Context(configuration) {{
Review comment:
Oh, yes. My mistake. It can be moved into Context.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547808056
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -246,6 +269,10 @@ private void requestNewWorker(WorkerResourceSpec workerResourceSpec) {
workerResourceSpec,
count,
exception);
+ if (recordWorkerFailure()) {
+ // if exceed failure rate try to slow down
+ tryResetWorkerCreationCoolDown();
+ }
Review comment:
The behavior is different in onWorkerTerminated as we only record failure events for workers that not yet registered. Probably just leave it as it is for now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r550417182
##########
File path: flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/ThresholdMeter.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+import java.time.Duration;
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/** A timestamp queue based threshold meter. */
+public class ThresholdMeter implements Meter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maxEventsPerInterval;
+ private final Duration interval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public ThresholdMeter(double maximumFailureRate, Duration interval) {
+ this(maximumFailureRate, interval, System::currentTimeMillis);
+ }
+
+ public ThresholdMeter(
Review comment:
As ManualClock needs extra dependeny on flink-core, I just remove the clock operation and simply relies on thread.sleep. The constructor can be removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-750038821
@xintongsong
Thanks for these valuable comments. Change the code according to most of your feedback. The only thing I keep is the failure rate interface as there is other use case. We may generalize it later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547856366
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRaterTest.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.ManualClock;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Test time stamp based failure rater.
+ */
+public class TimestampBasedFailureRaterTest {
Review comment:
Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-731982120
@HuangZhenQiu
ok, I have reopened the PR
Please, rebase it on master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r550381861
##########
File path: flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/ThresholdMeter.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+import java.time.Duration;
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/** A timestamp queue based threshold meter. */
+public class ThresholdMeter implements Meter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maxEventsPerInterval;
+ private final Duration interval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public ThresholdMeter(double maximumFailureRate, Duration interval) {
+ this(maximumFailureRate, interval, System::currentTimeMillis);
+ }
+
+ public ThresholdMeter(
+ double maxEventsPerInterval, Duration interval, Supplier<Long> customSupplier) {
+ this.maxEventsPerInterval = maxEventsPerInterval;
+ this.interval = interval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(currentTimeMillisSupplier.get());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ long timestamp = currentTimeMillisSupplier.get();
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(timestamp);
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getEventCountsRecentInterval() / (interval.toMillis() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ public void checkAgainstThreshold() throws ThresholdExceedException {
+ if (getEventCountsRecentInterval() >= maxEventsPerInterval) {
+ throw new ThresholdExceedException(
+ String.format(
+ "Maximum number of events %f is detected",
+ getEventCountsRecentInterval()));
+ }
+ }
+
+ private double getEventCountsRecentInterval() {
Review comment:
Return type should not be `double`
##########
File path: flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/ThresholdMeter.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+import java.time.Duration;
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/** A timestamp queue based threshold meter. */
+public class ThresholdMeter implements Meter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
Review comment:
The primitive type `double` is preferred.
##########
File path: flink-metrics/flink-metrics-core/src/test/java/org/apache/flink/metrics/ThresholdMeterTest.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+import org.apache.flink.metrics.ThresholdMeter.ThresholdExceedException;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import java.time.Duration;
+
+/** Test time stamp based threshold meter. */
+public class ThresholdMeterTest extends TestLogger {
Review comment:
I think we should also test against the contracts of `Meter`.
E.g., `markEvent`, `getRate`, `getCount`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -301,7 +353,25 @@ private boolean clearStateForWorker(ResourceID resourceId) {
return true;
}
- private void requestWorkerIfRequired() {
+ private void tryResetWorkerCreationCoolDown() {
+ if (workerCreationCoolDown.isDone()) {
+ log.info(
+ "Reaching max start worker failure rate. Will not retry creating worker in {}.",
+ workerCreationRetryInterval);
+ workerCreationCoolDown = new CompletableFuture<>();
+ getMainThreadExecutor()
+ .scheduleRunAsync(
+ () -> workerCreationCoolDown.complete(null),
+ workerCreationRetryInterval.getSize());
Review comment:
There's no guaranteed that `workerCreationRetryInterval.getSize` returns in milliseconds.
##########
File path: flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/ThresholdMeter.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+import java.time.Duration;
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/** A timestamp queue based threshold meter. */
+public class ThresholdMeter implements Meter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maxEventsPerInterval;
+ private final Duration interval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public ThresholdMeter(double maximumFailureRate, Duration interval) {
+ this(maximumFailureRate, interval, System::currentTimeMillis);
+ }
+
+ public ThresholdMeter(
+ double maxEventsPerInterval, Duration interval, Supplier<Long> customSupplier) {
+ this.maxEventsPerInterval = maxEventsPerInterval;
+ this.interval = interval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(currentTimeMillisSupplier.get());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ long timestamp = currentTimeMillisSupplier.get();
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(timestamp);
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getEventCountsRecentInterval() / (interval.toMillis() / MILLISECONDS_PER_SECOND);
Review comment:
Need to make sure `interval` is not `0`.
##########
File path: flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/ThresholdMeter.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+import java.time.Duration;
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/** A timestamp queue based threshold meter. */
+public class ThresholdMeter implements Meter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maxEventsPerInterval;
+ private final Duration interval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public ThresholdMeter(double maximumFailureRate, Duration interval) {
+ this(maximumFailureRate, interval, System::currentTimeMillis);
+ }
+
+ public ThresholdMeter(
Review comment:
Seems this constructor is no longer used.
We can remote this constructor, as well as `currentTimeMillisSupplier`.
##########
File path: flink-metrics/flink-metrics-core/src/test/java/org/apache/flink/metrics/ThresholdMeterTest.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+import org.apache.flink.metrics.ThresholdMeter.ThresholdExceedException;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import java.time.Duration;
+
+/** Test time stamp based threshold meter. */
+public class ThresholdMeterTest extends TestLogger {
+
+ @Test(expected = ThresholdExceedException.class)
+ public void testMaximumFailureCheck() {
+ ThresholdMeter rater = new ThresholdMeter(5, Duration.ofSeconds(10));
+
+ for (int i = 0; i < 6; i++) {
+ rater.markEvent();
+ }
+
+ rater.checkAgainstThreshold();
+ }
+
+ @Test(expected = ThresholdExceedException.class)
+ public void testRateRecordMultipleEvents() throws InterruptedException {
+ ThresholdMeter rater = new ThresholdMeter(5, Duration.ofMillis(500));
+
+ for (int i = 0; i < 3; i++) {
+ rater.markEvent(2);
+ Thread.sleep(150);
Review comment:
I think we can further decrease the sleep time, to reduce the time needed to run the tests.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -213,14 +235,33 @@ public void onPreviousAttemptWorkersRecovered(Collection<WorkerType> recoveredWo
}
}
+ /**
+ * Record failure number of worker in ResourceManagers. Return whether maximum failure rate is
+ * detected.
+ *
+ * @return whether should acquire new container/worker after the a stop interval
+ */
+ public boolean recordWorkerFailure() {
+ failureRater.markEvent();
+
+ try {
+ failureRater.checkAgainstThreshold();
+ } catch (ThresholdExceedException e) {
+ log.warn(e.getMessage() + " in resource manager failure rater.");
+ return true;
+ }
+
+ return false;
+ }
+
@Override
public void onWorkerTerminated(ResourceID resourceId, String diagnostics) {
if (clearStateForWorker(resourceId)) {
log.info(
"Worker {} is terminated. Diagnostics: {}",
resourceId.getStringWithMetadata(),
diagnostics);
- requestWorkerIfRequired();
+ recordWorkerFailureAndPauseWorkerCreationIfNeeded();
Review comment:
Incorrect condition. This will also record failures for previous attempt workers.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricNames.java
##########
@@ -62,6 +62,8 @@ private MetricNames() {}
public static final String CHECKPOINT_ALIGNMENT_TIME = "checkpointAlignmentTime";
public static final String CHECKPOINT_START_DELAY_TIME = "checkpointStartDelayNanos";
+ public static final String WORKER_FAILURE_RATE = "startWorkFailure" + SUFFIX_RATE;
Review comment:
This is never used. Seems the metrics is never registered.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -301,7 +353,25 @@ private boolean clearStateForWorker(ResourceID resourceId) {
return true;
}
- private void requestWorkerIfRequired() {
+ private void tryResetWorkerCreationCoolDown() {
+ if (workerCreationCoolDown.isDone()) {
+ log.info(
+ "Reaching max start worker failure rate. Will not retry creating worker in {}.",
+ workerCreationRetryInterval);
+ workerCreationCoolDown = new CompletableFuture<>();
+ getMainThreadExecutor()
Review comment:
For `ActiveResourceManager`, no need to call `getMainThreadExecutor`. Its super class `RpcEndpoint` already provided methods for scheduling runnable in the main thread.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -230,6 +271,12 @@ public void onError(Throwable exception) {
onFatalError(exception);
}
+ public static ThresholdMeter createFailureRater(Configuration configuration) {
+ double rate = configuration.getDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE);
+ Preconditions.checkArgument(rate > 0, "Failure rate should be larger than 0");
+ return new ThresholdMeter(rate, Duration.ofMinutes(1));
+ }
Review comment:
This should be moved to `ActiveResourceManagerFactory`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 99c02a09184876188e61b0149349ee0d9446660c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547734136
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/ResourceManagerOptions.java
##########
@@ -67,6 +67,33 @@
"for streaming workloads, which may fail if there are not enough slots. Note that this configuration option does not take " +
"effect for standalone clusters, where how many slots are allocated is not controlled by Flink.");
+ /**
+ * Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting subsequent worker
+ * requests until the failure rate falls below the maximum. It is to quickly catch external dependency caused
+ * workers failure and wait for retry interval before sending new request. Be default, -1.0 is set to disable the feature.
+ */
+ public static final ConfigOption<Double> MAXIMUM_WORKERS_FAILURE_RATE = ConfigOptions
+ .key("resourcemanager.maximum-workers-failure-rate")
+ .doubleType()
+ .defaultValue(-1.0)
Review comment:
It was the original value before combining the logic of the retry interval. I agree that a reasonable value such as 10/min should be the default value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-705393891
I think you can reopen it (at the bottom of the comment box to the left), once you start again working on it.
Let me me know if you do not have this button `Reopen and comment` and you start to work on, I will reopen the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 67e9e1e4c013b78eaee9fc9d215ef400bba5cc59 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309)
* c6cd36a8dfd854ecf63b2d248a4a28f889c873ec UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-731719725
@azagrebin
I don't have the button to reopen it. Would you please help me to reopen?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
}, {
"hash" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535",
"triggerID" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"triggerType" : "PUSH"
}, {
"hash" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11632",
"triggerID" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 8b920eb9c1f70c08f663154e15c60f6ed331394b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11632)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-705393891
I think you can reopen it (at the bottom of the comment box to the left), once you start again working on it.
Let me me know if you do not have this button `Reopen and comment` and you start to work on, I will reopen the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5c053117296cab1417bfbf98a0a8e63f48def5b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095)
* adf63fa7c5c73eecf4ae0237cb3f0fde2d616082 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
}, {
"hash" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535",
"triggerID" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"triggerType" : "PUSH"
}, {
"hash" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 50dd8154daca82c54bc3d0decc8e94b6b7cba8d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535)
* 8b920eb9c1f70c08f663154e15c60f6ed331394b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548938212
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/ThresholdMeter.java
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+/**
+ * Metric for measuring rate with Threshold.
+ */
+public interface ThresholdMeter extends Meter {
+
+ /**
+ * Check whether the current rate of events exceed threshold.
+ * If yes, throw ThresholdExceedException.
+ *
+ * @throws ThresholdExceedException
+ */
+ void checkAgainstThreshold() throws ThresholdExceedException;
Review comment:
Updated accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r550421330
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -213,14 +235,33 @@ public void onPreviousAttemptWorkersRecovered(Collection<WorkerType> recoveredWo
}
}
+ /**
+ * Record failure number of worker in ResourceManagers. Return whether maximum failure rate is
+ * detected.
+ *
+ * @return whether should acquire new container/worker after the a stop interval
+ */
+ public boolean recordWorkerFailure() {
+ failureRater.markEvent();
+
+ try {
+ failureRater.checkAgainstThreshold();
+ } catch (ThresholdExceedException e) {
+ log.warn(e.getMessage() + " in resource manager failure rater.");
+ return true;
+ }
+
+ return false;
+ }
+
@Override
public void onWorkerTerminated(ResourceID resourceId, String diagnostics) {
if (clearStateForWorker(resourceId)) {
log.info(
"Worker {} is terminated. Diagnostics: {}",
resourceId.getStringWithMetadata(),
diagnostics);
- requestWorkerIfRequired();
+ recordWorkerFailureAndPauseWorkerCreationIfNeeded();
Review comment:
Yes, we should record failure only for currentAttemptUnregisteredWorkers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548938104
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/execution/ThresholdExceedException.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.execution;
+
+/**
+ * Exception thrown when a threshold exceeds.
+ */
+public class ThresholdExceedException extends RuntimeException {
+
+ private static final long serialVersionUID = 241871312848349L;
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-704978874
@HuangZhenQiu
I am closing this PR for now. Please, reopen it if you decide to work on it again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r550418267
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricNames.java
##########
@@ -62,6 +62,8 @@ private MetricNames() {}
public static final String CHECKPOINT_ALIGNMENT_TIME = "checkpointAlignmentTime";
public static final String CHECKPOINT_START_DELAY_TIME = "checkpointStartDelayNanos";
+ public static final String WORKER_FAILURE_RATE = "startWorkFailure" + SUFFIX_RATE;
Review comment:
Yes, It is my mistake. There was a big code rebase after FLINK-20651 is landed. Will be more careful next time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 58784051e00030de475f361d314c1c0473ff2b8a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279)
* c703f433eefd8591026113e4ea848c6f01cf0dd6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-759073868
@xintongsong
Thanks for polishing the variable and function naming and comments. I am happy that the change is more readable now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-750703402
@HuangZhenQiu, thanks for updating the PR. I'll give it another pass asap.
Just to provide a quick response to your comment:
I think you are right that we should not use rate per minute for checking against the threshold. It looks like currently `getRatePerMinute` is only called for logging and tests. We probably do not need to introduce this in `Meter`.
* For logging, we can change `boolean exceedThreshold()` into `void checkAgainstThreashold() throws ThresholdExceedException`. In this way, we can get proper error message from the exception.
* For testing, I think we should verify the contract of `TimestampBasedFailureRater` rather than its internal states. To that end, if we no longer need the interface `getRatePerMinute`, we don't need to test against it neither.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-752320177
@HuangZhenQiu,
Sorry for the late response.
I don't think need to split the 100 LoC into `ThresholdMeter`, `TimestampBasedThresholdMeter`, and `FailureRater`.
On the contrary, I think we should not have the concepts `FailureRater` and `TimestampBasedThresholdMeter` at all.
Please see my other comment [here](https://github.com/apache/flink/pull/8952#discussion_r549927391).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e2d99f428b8c04df9f7709f9454898e2429d3cf0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* adf63fa7c5c73eecf4ae0237cb3f0fde2d616082 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-752880906
Thanks for the suggestion of a separate commit. I will follow the practice in future PRs. Happy new year.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 99c02a09184876188e61b0149349ee0d9446660c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243)
* f59351ab8f34947c26b357ef3f87e80a85996609 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547826360
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManagerTest.java
##########
@@ -378,15 +382,69 @@ public void testOnError() throws Exception {
}};
}
- private static class Context {
+ @Test
+ public void testWorkerCreationInterval() throws Exception {
+ Configuration configuration = new Configuration();
+ configuration.setDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE, 1);
+ configuration.setLong(ResourceManagerOptions.WORKER_CREATION_RETRY_INTERVAL,
+ WORKER_CREATION_INTERVAL.toMilliseconds());
+ new Context(configuration) {{
Review comment:
That's not true. The resource manager is created when `runTest` is called.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 67e9e1e4c013b78eaee9fc9d215ef400bba5cc59 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
}, {
"hash" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535",
"triggerID" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 50dd8154daca82c54bc3d0decc8e94b6b7cba8d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547804386
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/FailureRater.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.util.clock.Clock;
+
+/**
+ * Failure rate interface.
+ */
+public interface FailureRater extends Meter {
Review comment:
I think convenience of testing should not decide the contract of a production interface. However, we do want to implement the interface in a testing-friendly way. In this particular case, I believe mocking of time can be achieved in something like the following.
```
public class TimestampBasedFailureRater implements ThresholdMeter {
private final Supplier<Long> currentTimeMillisSupplier;
public TimestampBasedFailureRater() {
this.currentTimeMillisSupplier = System::currentTimeMillis;
}
@VisibleForTesting
public TimestampBasedFailureRater(Supplier<Long> customSupplier) {
this.currentTimeMillisSupplier = checkNotNull(customSupplier);
}
public void markEvent() {
long eventTimeStamp = currentTimeMillisSupplier.get();
}
}
```
As for rate per minute, we can easily extend `Meter` with a `getRatePerMinute`.
```
public interface Meter extend Metric {
double getRate();
default double getRatePerMinute {
return getRate() * 60.0;
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 67e9e1e4c013b78eaee9fc9d215ef400bba5cc59 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309)
* c6cd36a8dfd854ecf63b2d248a4a28f889c873ec Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c703f433eefd8591026113e4ea848c6f01cf0dd6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072)
* e2d99f428b8c04df9f7709f9454898e2429d3cf0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 70b43a088878601fa074245f88800b163641d6f0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426)
* fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548938183
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/FailureRaterUtil.java
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.ResourceManagerOptions;
+import org.apache.flink.util.Preconditions;
+
+/**
+ * Failure rate util.
+ */
+public class FailureRaterUtil {
Review comment:
Agree. It should be moved to TimestampBasedFailureRater after NoFailreRater is removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 7cffa19fd89147eb546988f22f2f98c911e044b4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273)
* 17290d66d3ad93815bdfea6343857b62ed182772 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 70b43a088878601fa074245f88800b163641d6f0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426)
* fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
}, {
"hash" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493)
* 50dd8154daca82c54bc3d0decc8e94b6b7cba8d6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-739733015
Thanks for the PR, @HuangZhenQiu.
Before diving into the implementation details, I'd like to have some more high level discussions, concerning the problem we want to solve and the expected behaviors. I'll comment the details on the jira ticket.
Sorry for the pulling back. I known there has been a long journey for this effort.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547770638
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.ArrayDeque;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements FailureRater {
+ private static final int DEFAULT_TIMESTAMP_SIZE = 300;
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final int maximumFailureRate;
+ private final Time failureInterval;
+ private final ArrayDeque<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(int maximumFailureRate, Time failureInterval) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>(maximumFailureRate > 0 ? maximumFailureRate : DEFAULT_TIMESTAMP_SIZE);
+ }
+
+ @Override
+ public void markEvent() {
+ markEvent(System.currentTimeMillis());
+ }
+
+ @Override
+ public void markEvent(long n) {
+ failureTimestamps.add(n);
+ failureCounter++;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() / (failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public boolean exceedsFailureRate() {
+ return getCurrentFailureRate() >= maximumFailureRate;
+ }
+
+ @Override
+ public void markFailure(Clock clock) {
+ failureTimestamps.add(clock.absoluteTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public double getCurrentFailureRate() {
+ Long currentTimeStamp = System.currentTimeMillis();
+ while (!failureTimestamps.isEmpty() &&
+ currentTimeStamp - failureTimestamps.peek() > failureInterval.toMilliseconds()) {
+ failureTimestamps.remove();
+ }
+
+ return failureTimestamps.size();
+ }
+
+ public Time getFailureInterval() {
+ return failureInterval;
+ }
Review comment:
Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548939285
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
Review comment:
There actually a three-layer concept here. ThresholdMeter is a Meter that has a checkAgainstThreshold interface. The TimestampBasedThresholdMeter is a concrete implementation. The failure rate is actually a customized TimestampBasedThresholdMeter, as we put the fromConfiguration and exception message in the class. I am good with split into three layers. But as it is the only use case, probably it will be better to keep it as it is?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-739733015
Thanks for the PR, @HuangZhenQiu.
Before dive into the implementation details, I'd like to have some more high level discussions, concerning the problem we want to solve and the expected behaviors. I'll comment the details on the jira ticket.
Sorry for the pulling back. I known there has been a long journey for this effort.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c703f433eefd8591026113e4ea848c6f01cf0dd6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547883286
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/FailureRater.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.util.clock.Clock;
+
+/**
+ * Failure rate interface.
+ */
+public interface FailureRater extends Meter {
Review comment:
Agree. Added ThresholdMeter into flink-metircs-core module, and changed TimestampBasedFailureRater accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547770563
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.ArrayDeque;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements FailureRater {
+ private static final int DEFAULT_TIMESTAMP_SIZE = 300;
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final int maximumFailureRate;
+ private final Time failureInterval;
+ private final ArrayDeque<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(int maximumFailureRate, Time failureInterval) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>(maximumFailureRate > 0 ? maximumFailureRate : DEFAULT_TIMESTAMP_SIZE);
+ }
+
+ @Override
+ public void markEvent() {
+ markEvent(System.currentTimeMillis());
+ }
+
+ @Override
+ public void markEvent(long n) {
+ failureTimestamps.add(n);
+ failureCounter++;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() / (failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public boolean exceedsFailureRate() {
+ return getCurrentFailureRate() >= maximumFailureRate;
+ }
+
+ @Override
+ public void markFailure(Clock clock) {
+ failureTimestamps.add(clock.absoluteTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public double getCurrentFailureRate() {
+ Long currentTimeStamp = System.currentTimeMillis();
+ while (!failureTimestamps.isEmpty() &&
+ currentTimeStamp - failureTimestamps.peek() > failureInterval.toMilliseconds()) {
+ failureTimestamps.remove();
+ }
+
+ return failureTimestamps.size();
Review comment:
Mentioned above for testing purposes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-752331980
@xintongsong
In the current implementation, I choose to use the class name TimestampBasedFailureRater rather than ThresholdMeter because there is a container failure related error message and config key are used in the class. I think it is not generic enough to name it as ThresholdMeter. For the formatting rules. I used spotless to format the code after rebasing the master. The code should already conform to the rule proposed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* f59351ab8f34947c26b357ef3f87e80a85996609 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262)
* 7cffa19fd89147eb546988f22f2f98c911e044b4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-750402556
@xintongsong
For your last three comments, I updated accordingly. The last concern from my side is that the ratePerMinute in Meter is kind of different from the real currentFailureRate. For example, if there are 3 failures in the first 1 second, the rate per minute will return 3 * 60 = 180, but the value we use to compare with the threshold is the value seen since the last one minute which is still 3. Thus, the alert threshold will be different from the maximum failure rate specified in config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r550417531
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -301,7 +353,25 @@ private boolean clearStateForWorker(ResourceID resourceId) {
return true;
}
- private void requestWorkerIfRequired() {
+ private void tryResetWorkerCreationCoolDown() {
+ if (workerCreationCoolDown.isDone()) {
+ log.info(
+ "Reaching max start worker failure rate. Will not retry creating worker in {}.",
+ workerCreationRetryInterval);
+ workerCreationCoolDown = new CompletableFuture<>();
+ getMainThreadExecutor()
Review comment:
Ok, just found the function with a different parameter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547800842
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRaterTest.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.ManualClock;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Test time stamp based failure rater.
+ */
+public class TimestampBasedFailureRaterTest {
+
+ @Test
+ public void testMaximumFailureCheck() {
+ TimestampBasedFailureRater rater = new TimestampBasedFailureRater(5, Time.of(10, TimeUnit.SECONDS));
+
+ for (int i = 0; i < 6; i++) {
+ rater.markEvent();
+ }
+
+ Assert.assertEquals(6.0, rater.getCurrentFailureRate(), 0.01);
+ Assert.assertTrue(rater.exceedsFailureRate());
+ }
+
+ @Test
+ public void testMovingRate() throws InterruptedException {
+ FailureRater rater = new TimestampBasedFailureRater(5, Time.of(500, TimeUnit.MILLISECONDS));
+
+ ManualClock manualClock = new ManualClock();
+ manualClock.advanceTime(System.currentTimeMillis(), TimeUnit.MILLISECONDS);
+ for (int i = 0; i < 6; i++) {
+ rater.markFailure(manualClock);
+ manualClock.advanceTime(150, TimeUnit.MILLISECONDS);
+ Thread.sleep(150);
+ }
+
+ Assert.assertEquals(3.0, rater.getCurrentFailureRate(), 0.01);
+ Assert.assertFalse(rater.exceedsFailureRate());
+ }
+}
Review comment:
Agree. Added test cases for markEvent(int n).
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManagerTest.java
##########
@@ -378,15 +382,69 @@ public void testOnError() throws Exception {
}};
}
- private static class Context {
+ @Test
+ public void testWorkerCreationInterval() throws Exception {
+ Configuration configuration = new Configuration();
+ configuration.setDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE, 1);
+ configuration.setLong(ResourceManagerOptions.WORKER_CREATION_RETRY_INTERVAL,
+ WORKER_CREATION_INTERVAL.toMilliseconds());
+ new Context(configuration) {{
Review comment:
Revised.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-747899670
@xintongsong
Thanks for reviewing the PR. I replied to your comments on the Jira ticket. Once we have an agreement on the implementation, I will revise the PR accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* c6cd36a8dfd854ecf63b2d248a4a28f889c873ec Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 70b43a088878601fa074245f88800b163641d6f0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r550416903
##########
File path: flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/ThresholdMeter.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+import java.time.Duration;
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/** A timestamp queue based threshold meter. */
+public class ThresholdMeter implements Meter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maxEventsPerInterval;
+ private final Duration interval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public ThresholdMeter(double maximumFailureRate, Duration interval) {
+ this(maximumFailureRate, interval, System::currentTimeMillis);
+ }
+
+ public ThresholdMeter(
+ double maxEventsPerInterval, Duration interval, Supplier<Long> customSupplier) {
+ this.maxEventsPerInterval = maxEventsPerInterval;
+ this.interval = interval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(currentTimeMillisSupplier.get());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ long timestamp = currentTimeMillisSupplier.get();
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(timestamp);
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getEventCountsRecentInterval() / (interval.toMillis() / MILLISECONDS_PER_SECOND);
Review comment:
Agree. Checked in the constructor.
##########
File path: flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/ThresholdMeter.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+import java.time.Duration;
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/** A timestamp queue based threshold meter. */
+public class ThresholdMeter implements Meter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maxEventsPerInterval;
+ private final Duration interval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public ThresholdMeter(double maximumFailureRate, Duration interval) {
+ this(maximumFailureRate, interval, System::currentTimeMillis);
+ }
+
+ public ThresholdMeter(
+ double maxEventsPerInterval, Duration interval, Supplier<Long> customSupplier) {
+ this.maxEventsPerInterval = maxEventsPerInterval;
+ this.interval = interval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(currentTimeMillisSupplier.get());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ long timestamp = currentTimeMillisSupplier.get();
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(timestamp);
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getEventCountsRecentInterval() / (interval.toMillis() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ public void checkAgainstThreshold() throws ThresholdExceedException {
+ if (getEventCountsRecentInterval() >= maxEventsPerInterval) {
+ throw new ThresholdExceedException(
+ String.format(
+ "Maximum number of events %f is detected",
+ getEventCountsRecentInterval()));
+ }
+ }
+
+ private double getEventCountsRecentInterval() {
Review comment:
Done.
##########
File path: flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/ThresholdMeter.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+import java.time.Duration;
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/** A timestamp queue based threshold meter. */
+public class ThresholdMeter implements Meter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r550417341
##########
File path: flink-metrics/flink-metrics-core/src/test/java/org/apache/flink/metrics/ThresholdMeterTest.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics;
+
+import org.apache.flink.metrics.ThresholdMeter.ThresholdExceedException;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import java.time.Duration;
+
+/** Test time stamp based threshold meter. */
+public class ThresholdMeterTest extends TestLogger {
Review comment:
The test on rate per second will be flaky. But added assert on getCount on tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-751135123
@xintongsong
Thanks for the suggestion. It is a feasible way to make the rate value consistent in both log and metrics. I updated accordingly. Merry Christmas.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547752610
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/FailureRater.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.util.clock.Clock;
+
+/**
+ * Failure rate interface.
+ */
+public interface FailureRater extends Meter {
Review comment:
ThresholdMeter should be good enough, but do you see any other usage for ThresholdMeter. Probably, keep the naming here for current usage. We can change it if there is a need later. For the interface, I guess we can keep exceedsFailureRate and markFailure.
1) As you pointed out, markEvent takes the count of occurrence as input. But we need a timestamp as parameter here for implementation and test purposes. (please check the TimestampBasedFailureRaterTest).
2) For getCurrentFailureRate, we return the rate per minute. In the meter interface, getRate returns the rate per second. So getCurrentFailureRate is for the internal logic, getRate is for metrics reporting. Should we just keep it here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-735044796
@tillrohrmann @zentol @azagrebin Rebased master. Please review it when you have time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547769505
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/NoFailureRater.java
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.metrics.MeterView;
+import org.apache.flink.util.clock.Clock;
+
+/**
+ * No failure rater only records to the per second rate for metrics reporting. Thus, its functionality falls back to
+ * a MeterView.
+ */
+public class NoFailureRater extends MeterView implements FailureRater {
+
+ public NoFailureRater(int timeSpanInSeconds) {
+ super(timeSpanInSeconds);
+ }
+
+ @Override
+ public boolean exceedsFailureRate() {
+ return false;
+ }
+
+ @Override
+ public void markFailure(Clock clock) {
+ markEvent(clock.absoluteTimeMillis());
Review comment:
Good catch. Revised accordingly. We need clock in the interface, so that It is easier for us to simulate the time changes in test cases. Please check TimestampBasedFailureRaterTest.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.ArrayDeque;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements FailureRater {
+ private static final int DEFAULT_TIMESTAMP_SIZE = 300;
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final int maximumFailureRate;
+ private final Time failureInterval;
+ private final ArrayDeque<Long> failureTimestamps;
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-737618805
I'll try to find some time for this next week. Right now I'm a bit underwater.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c703f433eefd8591026113e4ea848c6f01cf0dd6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072)
* e2d99f428b8c04df9f7709f9454898e2429d3cf0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548814271
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/execution/ThresholdExceedException.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.execution;
+
+/**
+ * Exception thrown when a threshold exceeds.
+ */
+public class ThresholdExceedException extends RuntimeException {
Review comment:
Why is this class in the path `runtime/execution`?
I think we should make it an inner class of `ThresholdMeter`, given that the two classes are closely related.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/execution/ThresholdExceedException.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.execution;
+
+/**
+ * Exception thrown when a threshold exceeds.
+ */
+public class ThresholdExceedException extends RuntimeException {
+
+ private static final long serialVersionUID = 241871312848349L;
Review comment:
`serialVersionUID` should start from `1L`.
https://flink.apache.org/contributing/code-style-and-quality-java.html#java-serialization
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval) {
+ this(maximumFailureRate, failureInterval, System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval, Supplier<Long> customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(System.currentTimeMillis());
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() / (failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public void checkAgainstThreshold() throws ThresholdExceedException {
+ if (getCurrentFailureRate() >= maximumFailureRate) {
+ throw new ThresholdExceedException(String.format("Maximum number of failed workers %f"
+ + " is detected in Resource Manager", getCurrentFailureRate()));
+ }
+ }
+
+ private double getCurrentFailureRate() {
+ Long currentTimeStamp = System.currentTimeMillis();
Review comment:
```suggestion
Long currentTimeStamp = currentTimeMillisSupplier.ge();
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval) {
+ this(maximumFailureRate, failureInterval, System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval, Supplier<Long> customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(System.currentTimeMillis());
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() / (failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public void checkAgainstThreshold() throws ThresholdExceedException {
+ if (getCurrentFailureRate() >= maximumFailureRate) {
+ throw new ThresholdExceedException(String.format("Maximum number of failed workers %f"
+ + " is detected in Resource Manager", getCurrentFailureRate()));
+ }
+ }
+
+ private double getCurrentFailureRate() {
+ Long currentTimeStamp = System.currentTimeMillis();
+ while (!failureTimestamps.isEmpty() &&
+ currentTimeStamp - failureTimestamps.peek() > failureInterval.toMilliseconds()) {
+ failureTimestamps.remove();
+ }
+
+ return failureTimestamps.size();
Review comment:
What is being returned is the count of events occurred within the interval. It's not a rate.
Maybe rename the method to `getFailuresCountsRecentInterval`?
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/ResourceManagerOptions.java
##########
@@ -67,6 +69,33 @@
"for streaming workloads, which may fail if there are not enough slots. Note that this configuration option does not take " +
"effect for standalone clusters, where how many slots are allocated is not controlled by Flink.");
+ /**
+ * Defines the maximum number of worker (YARN / Mesos / Kubernetes) failures per minute before rejecting subsequent worker
+ * requests until the failure rate falls below the maximum. It is to quickly catch external dependency caused
+ * workers failure and wait for retry interval before sending new request. By default, the value is set to 10/min.
+ */
+ public static final ConfigOption<Double> MAXIMUM_WORKERS_FAILURE_RATE = ConfigOptions
+ .key("resourcemanager.start-worker.max-failure-rate")
+ .doubleType()
+ .defaultValue(10.0)
+ .withDescription("Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting" +
+ " subsequent worker requests until the failure rate falls below the maximum. It is to quickly catch" +
+ " external dependency caused workers failure and terminate job accordingly." +
+ " By default, the value is set to 10/min.");
+
+ /**
+ * Defines the worker creation interval in milliseconds. In case of worker creation failures, we should wait for an interval before
+ * trying to create new workers when the failure rate exceeds. Otherwise, ActiveResourceManager will always re-requesting
+ * the worker, which keeps the main thread busy.
+ */
+ public static final ConfigOption<Duration> WORKER_CREATION_RETRY_INTERVAL = ConfigOptions
+ .key("resourcemanager.start-worker.retry-interval")
+ .durationType()
+ .defaultValue(Duration.ofMillis(30))
Review comment:
`30ms` might be too small.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
Review comment:
I think `failurerate` is a bit too detailed to be a direct child package under `runtime`.
Currently, codes in this package are only used by the active resource manager.
If `FailureRaterUtil` and `ThresholdMeter` are removed, we don't even need a package. We can put `TimestampBasedFailureRater` directly in package `resourcemanager.active`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/FailureRaterUtil.java
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.ResourceManagerOptions;
+import org.apache.flink.util.Preconditions;
+
+/**
+ * Failure rate util.
+ */
+public class FailureRaterUtil {
Review comment:
It's not necessary to introduce this util class.
We can simply add a static factory method in `TimestampBasedFailureRater`.
```
public class TimestampBasedFailureRater {
public static TimestampBasedFailureRater fromConfiguration(Configuration config) {
...
}
}
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/ThresholdMeter.java
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+/**
+ * Metric for measuring rate with Threshold.
+ */
+public interface ThresholdMeter extends Meter {
+
+ /**
+ * Check whether the current rate of events exceed threshold.
+ * If yes, throw ThresholdExceedException.
+ *
+ * @throws ThresholdExceedException
+ */
+ void checkAgainstThreshold() throws ThresholdExceedException;
Review comment:
I noticed that this interface is only implemented by `TimestampBasedFailureRater`, and only called by `ActiveResourceManager` and `TimestampBasedFailureRaterTest`.
I start to wonder, do we really need this interface? Sorry for the back and forth. I think we made `ActiveResourceManager` only aware of `ThresholdMeter` (or `FailureRater` previously) because we had different implementations of this interface. Now since we no longer have `NoFailureRater`, we probably can get rid of this interface, making `ActiveResourceManager` to use `TimestampBasedFailureRater` directly.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
Review comment:
Not sure about the class name `TimestampBasedFailureRater`. I believe this is named with respect to the previous interface `FailureRater` and the other implementation `NoFailureRater`.
Actually, I think this implementation class should be named `ThresholdMeter`, which implements `Meter` directly. Imagine replacing all the keyword "failure" with "event", you should find this class becoming a general implementation of `Meter` with `checkAgainstThreshold` being the only additional function.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval) {
+ this(maximumFailureRate, failureInterval, System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval, Supplier<Long> customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
Review comment:
```suggestion
failureTimestamps.add(currentTimeMillisSupplier.get());
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval) {
+ this(maximumFailureRate, failureInterval, System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval, Supplier<Long> customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(System.currentTimeMillis());
Review comment:
```suggestion
failureTimestamps.add(currentTimeMillisSupplier.ge());
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval) {
+ this(maximumFailureRate, failureInterval, System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval, Supplier<Long> customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(System.currentTimeMillis());
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() / (failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public void checkAgainstThreshold() throws ThresholdExceedException {
+ if (getCurrentFailureRate() >= maximumFailureRate) {
Review comment:
`getCurrentFailureRate` returns count of the event in the recent `failureInterval`, while `maximumFailureRate` is defined as "failures per minute" per the config option description.
The correctness of this check strongly depends on `failureInterval` being 1min.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricNames.java
##########
@@ -67,6 +67,7 @@ private MetricNames() {
public static final String CHECKPOINT_ALIGNMENT_TIME = "checkpointAlignmentTime";
public static final String CHECKPOINT_START_DELAY_TIME = "checkpointStartDelayNanos";
+ public static final String WORKER_FAILURE_RATE = "workFailureRate";
Review comment:
```suggestion
public static final String WORKER_FAILURE_RATE = "startWorkFailure" + SUFFIX_RATE;
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
##########
@@ -108,6 +111,8 @@
public static final String RESOURCE_MANAGER_NAME = "resourcemanager";
+ protected final Logger log = LoggerFactory.getLogger(getClass());
Review comment:
Unnecessary. The logger is already defined in `RpcEndpoint`, which is a super class of `ResourceManager.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
Review comment:
And if we turn this class into a general implementation of `Meter`, we may move it to the package `org.apache.flink.metrics`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval) {
+ this(maximumFailureRate, failureInterval, System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval, Supplier<Long> customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(System.currentTimeMillis());
Review comment:
And we should avoid calling `System.currentTimeMillis()` (or `currentTimeMillisSupplier.ge()`) for `n` times. The `n` events should have the same timestamp.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval) {
+ this(maximumFailureRate, failureInterval, System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval, Supplier<Long> customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(System.currentTimeMillis());
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() / (failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public void checkAgainstThreshold() throws ThresholdExceedException {
+ if (getCurrentFailureRate() >= maximumFailureRate) {
Review comment:
I see several different *rates* here, and I think we should make them clearly defined.
* `getRate()` returns a per-second rate, which is clearly defined in the interface JavaDoc. We can also add a comment to remind people of that.
* What is the definition of `maximumFailureRate`? We probably should name it `maxFailuresPerMinute`, `maxFailuresPerSecond` or `maxFailuresPerFailureInterval`.
* What is the definition of `getCurrentFailureRate`? We probably should name it `getFailuresCountsRecentInterval`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricNames.java
##########
@@ -67,6 +67,7 @@ private MetricNames() {
public static final String CHECKPOINT_ALIGNMENT_TIME = "checkpointAlignmentTime";
public static final String CHECKPOINT_START_DELAY_TIME = "checkpointStartDelayNanos";
+ public static final String WORKER_FAILURE_RATE = "workFailureRate";
Review comment:
Not sure if per-second is a good granularity for this metric.
Maybe it's not a big deal since people can always convert it to other granularity.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547805867
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -212,6 +227,9 @@ public void onPreviousAttemptWorkersRecovered(Collection<WorkerType> recoveredWo
@Override
public void onWorkerTerminated(ResourceID resourceId, String diagnostics) {
+ if (recordWorkerFailure()) {
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147257697) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798)
* 58784051e00030de475f361d314c1c0473ff2b8a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r549959072
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
Review comment:
I believe a clear abstraction of concepts should be prior to the implementation details (what the error messages say, how the factories should look like, etc). Compared with `Meter`, conceptually the only additional function this implementation class provides is `checkAgainstThreshold`.
Concerning the implementation issues you mentioned, they should be adjusted with respect to the underlying concepts.
* `checkAgainstThreshold` can only say that current rate `x` has reached the threshold `y`, and we can mention that the rate is for starting container failure in `ActiveResourceManager` for logging.
* We can change `fromConfiguration(Configuration config)` to `withMinuteThreshold(double threshold)` and parse the configuration option in `ActiveResourceManagerFactory`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* d58927642909f50571ed6242605aac564e074f89 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349)
* 70b43a088878601fa074245f88800b163641d6f0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547112777
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/ResourceManagerOptions.java
##########
@@ -67,6 +67,33 @@
"for streaming workloads, which may fail if there are not enough slots. Note that this configuration option does not take " +
"effect for standalone clusters, where how many slots are allocated is not controlled by Flink.");
+ /**
+ * Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting subsequent worker
+ * requests until the failure rate falls below the maximum. It is to quickly catch external dependency caused
+ * workers failure and wait for retry interval before sending new request. Be default, -1.0 is set to disable the feature.
+ */
+ public static final ConfigOption<Double> MAXIMUM_WORKERS_FAILURE_RATE = ConfigOptions
+ .key("resourcemanager.maximum-workers-failure-rate")
+ .doubleType()
+ .defaultValue(-1.0)
+ .withDescription("Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting" +
+ " subsequent worker requests until the failure rate falls below the maximum. It is to quickly catch" +
+ " external dependency caused workers failure and terminate job accordingly." +
+ " Be default, -1.0 is set to disable the feature.");
+
+ /**
+ * Defines the worker creation interval in milliseconds. In case of worker creation failures, we should wait for an interval before
+ * trying to create new workers when the failure rate exceeds. Otherwise, ActiveResourceManager will always re-requesting
+ * the worker, which keeps the main thread busy.
+ */
+ public static final ConfigOption<Long> WORKER_CREATION_RETRY_INTERVAL = ConfigOptions
+ .key("resourcemanager.workers-creation-retry-interval")
+ .longType()
Review comment:
`durationType()` is preferred for intervals.
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/ResourceManagerOptions.java
##########
@@ -67,6 +67,33 @@
"for streaming workloads, which may fail if there are not enough slots. Note that this configuration option does not take " +
"effect for standalone clusters, where how many slots are allocated is not controlled by Flink.");
+ /**
+ * Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting subsequent worker
+ * requests until the failure rate falls below the maximum. It is to quickly catch external dependency caused
+ * workers failure and wait for retry interval before sending new request. Be default, -1.0 is set to disable the feature.
+ */
+ public static final ConfigOption<Double> MAXIMUM_WORKERS_FAILURE_RATE = ConfigOptions
+ .key("resourcemanager.maximum-workers-failure-rate")
+ .doubleType()
+ .defaultValue(-1.0)
+ .withDescription("Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting" +
+ " subsequent worker requests until the failure rate falls below the maximum. It is to quickly catch" +
+ " external dependency caused workers failure and terminate job accordingly." +
+ " Be default, -1.0 is set to disable the feature.");
+
+ /**
+ * Defines the worker creation interval in milliseconds. In case of worker creation failures, we should wait for an interval before
+ * trying to create new workers when the failure rate exceeds. Otherwise, ActiveResourceManager will always re-requesting
+ * the worker, which keeps the main thread busy.
+ */
+ public static final ConfigOption<Long> WORKER_CREATION_RETRY_INTERVAL = ConfigOptions
+ .key("resourcemanager.workers-creation-retry-interval")
+ .longType()
+ .defaultValue(3000L)
+ .withDescription("Defines the worker creation interval in milliseconds. In case of worker creation failures,"
+ + " we should wait for an interval before trying to create new workers when the failure rate exceeds."
+ + " Otherwise, ActiveResourceManager will always re-requesting the worker, which keeps the main thread busy.");
+
Review comment:
I would suggest the following as keys for these configuration options.
* `resourcemanager.start-worker.max-failure-rate`
* `resourcemanager.start-worker.retry-internal`
The common prefix suggests these two options are related. Also, "start worker failure" is more descriptive since "worker failure" can also mean workers failed after registration.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.ArrayDeque;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements FailureRater {
+ private static final int DEFAULT_TIMESTAMP_SIZE = 300;
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final int maximumFailureRate;
+ private final Time failureInterval;
+ private final ArrayDeque<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(int maximumFailureRate, Time failureInterval) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>(maximumFailureRate > 0 ? maximumFailureRate : DEFAULT_TIMESTAMP_SIZE);
+ }
+
+ @Override
+ public void markEvent() {
+ markEvent(System.currentTimeMillis());
Review comment:
Same here. I think we are not using `markEvent` correctly.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.ArrayDeque;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements FailureRater {
+ private static final int DEFAULT_TIMESTAMP_SIZE = 300;
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final int maximumFailureRate;
+ private final Time failureInterval;
+ private final ArrayDeque<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(int maximumFailureRate, Time failureInterval) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>(maximumFailureRate > 0 ? maximumFailureRate : DEFAULT_TIMESTAMP_SIZE);
+ }
+
+ @Override
+ public void markEvent() {
+ markEvent(System.currentTimeMillis());
+ }
+
+ @Override
+ public void markEvent(long n) {
+ failureTimestamps.add(n);
+ failureCounter++;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() / (failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public boolean exceedsFailureRate() {
+ return getCurrentFailureRate() >= maximumFailureRate;
+ }
+
+ @Override
+ public void markFailure(Clock clock) {
+ failureTimestamps.add(clock.absoluteTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public double getCurrentFailureRate() {
+ Long currentTimeStamp = System.currentTimeMillis();
+ while (!failureTimestamps.isEmpty() &&
+ currentTimeStamp - failureTimestamps.peek() > failureInterval.toMilliseconds()) {
+ failureTimestamps.remove();
+ }
+
+ return failureTimestamps.size();
Review comment:
The calculation logic strongly depends on that failures in queue are strictly ordered by time.
To ensure that, we should not expose the argument `clock` in `markFailure`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
##########
@@ -860,6 +875,24 @@ private void clearStateInternal() {
clearStateFuture = clearStateAsync();
}
+ /**
+ * Record failure number of worker in ResourceManagers. If maximum failure rate is detected,
+ * then cancel all pending requests.
+ *
+ * @return whether should acquire new container/worker after the a stop interval
+ */
+ @VisibleForTesting
+ public boolean recordWorkerFailure() {
Review comment:
This should also belong to `ActiveResourceManager`.
Also, `@VisibleForTesting` should be removed.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -259,6 +286,22 @@ private void requestNewWorker(WorkerResourceSpec workerResourceSpec) {
}));
}
+ private void tryResetWorkerCreationCoolDown() {
+ if (workerCreationCoolDown.isDone()) {
+ log.info("Worker creation failed. Will not retry creating worker in {}.", workerCreationRetryInterval);
Review comment:
```suggestion
log.info("Reaching max start worker failure rate. Will not retry creating worker in {}.", workerCreationRetryInterval);
```
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/ResourceManagerOptions.java
##########
@@ -67,6 +67,33 @@
"for streaming workloads, which may fail if there are not enough slots. Note that this configuration option does not take " +
"effect for standalone clusters, where how many slots are allocated is not controlled by Flink.");
+ /**
+ * Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting subsequent worker
+ * requests until the failure rate falls below the maximum. It is to quickly catch external dependency caused
+ * workers failure and wait for retry interval before sending new request. Be default, -1.0 is set to disable the feature.
+ */
+ public static final ConfigOption<Double> MAXIMUM_WORKERS_FAILURE_RATE = ConfigOptions
+ .key("resourcemanager.maximum-workers-failure-rate")
+ .doubleType()
+ .defaultValue(-1.0)
Review comment:
I'm not sure about disabling this feature by default. I suspect not many people would be aware of the feature it's disabled by default. It might make sense to enable the feature a reasonable default rate, e.g., 10/min.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/NoFailureRater.java
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.metrics.MeterView;
+import org.apache.flink.util.clock.Clock;
+
+/**
+ * No failure rater only records to the per second rate for metrics reporting. Thus, its functionality falls back to
+ * a MeterView.
+ */
+public class NoFailureRater extends MeterView implements FailureRater {
+
+ public NoFailureRater(int timeSpanInSeconds) {
+ super(timeSpanInSeconds);
+ }
+
+ @Override
+ public boolean exceedsFailureRate() {
+ return false;
+ }
+
+ @Override
+ public void markFailure(Clock clock) {
+ markEvent(clock.absoluteTimeMillis());
Review comment:
I think this should not work.
According to JavaDoc of `Meter#markEvent`, the argument represents number of events occurred, not the timestamp when the event occurred.
I think we don't really need the argument `clock` in the interface. Shouldn't the event always be recorded with the current timestamp?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.ArrayDeque;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements FailureRater {
+ private static final int DEFAULT_TIMESTAMP_SIZE = 300;
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final int maximumFailureRate;
+ private final Time failureInterval;
+ private final ArrayDeque<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(int maximumFailureRate, Time failureInterval) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>(maximumFailureRate > 0 ? maximumFailureRate : DEFAULT_TIMESTAMP_SIZE);
+ }
+
+ @Override
+ public void markEvent() {
+ markEvent(System.currentTimeMillis());
+ }
+
+ @Override
+ public void markEvent(long n) {
+ failureTimestamps.add(n);
+ failureCounter++;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() / (failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public boolean exceedsFailureRate() {
+ return getCurrentFailureRate() >= maximumFailureRate;
+ }
+
+ @Override
+ public void markFailure(Clock clock) {
+ failureTimestamps.add(clock.absoluteTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public double getCurrentFailureRate() {
+ Long currentTimeStamp = System.currentTimeMillis();
+ while (!failureTimestamps.isEmpty() &&
+ currentTimeStamp - failureTimestamps.peek() > failureInterval.toMilliseconds()) {
+ failureTimestamps.remove();
+ }
+
+ return failureTimestamps.size();
+ }
+
+ public Time getFailureInterval() {
+ return failureInterval;
+ }
Review comment:
Unused
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/FailureRater.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.util.clock.Clock;
+
+/**
+ * Failure rate interface.
+ */
+public interface FailureRater extends Meter {
Review comment:
This interface has nothing specific to *failures*.
We might consider to make it more general and reusable, something like `ThresholdMeter`.
We can reuse `Meter#markEvent` for `markFailure` and `Meter#getRate` for `getCurrentFailureRate`. Only `exceedsFailureRate` needs to be added.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/FailureRaterUtil.java
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.ResourceManagerOptions;
+
+/**
+ * Failure rate util.
+ */
+public class FailureRaterUtil {
+
+ public static FailureRater createFailureRater(
+ Configuration configuration) {
+ double rate = configuration.getDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE);
+ if (rate < 0) {
+ return new NoFailureRater(60);
+ } else {
+ int maximumRate = Double.valueOf(rate).intValue();
+ return new TimestampBasedFailureRater(maximumRate < rate ? maximumRate + 1 : maximumRate, Time.minutes(1));
+ }
Review comment:
I don't really think we need this util class.
Not sure if we need a two different raters. `NoFailureRater` might be a bit light weighted compared to `TimestampBasedFailureRater`. But I wonder how significant the difference could be. Would it worth to complicates the codes and maintain two different raters?
Also, I think we can convert the max rate from double to integer inside `TimestampBasedFailureRater`. It's a bit confusing to see an integer type *rate* in the constructor argument list.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.ArrayDeque;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements FailureRater {
+ private static final int DEFAULT_TIMESTAMP_SIZE = 300;
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final int maximumFailureRate;
+ private final Time failureInterval;
+ private final ArrayDeque<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(int maximumFailureRate, Time failureInterval) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>(maximumFailureRate > 0 ? maximumFailureRate : DEFAULT_TIMESTAMP_SIZE);
Review comment:
Why do we make the initial capacity `maximumFailureRate`?
There's no guarantee that the queue sizes won't exceed `maximumFailureRate`. New failures can still be recorded when the max rate is reached.
On the other hand, in most cases where failures do not happen a lot, this queue should be almost empty.
The community's code style guidelines suggest "Set the initial capacity for a collection only if there is a good proven reason".
https://flink.apache.org/contributing/code-style-and-quality-java.html#collections
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
##########
@@ -143,6 +150,8 @@
private final ResourceManagerMetricGroup resourceManagerMetricGroup;
+ protected final FailureRater failureRater;
Review comment:
I think we should only introduce this for the active resource managers.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.ArrayDeque;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements FailureRater {
+ private static final int DEFAULT_TIMESTAMP_SIZE = 300;
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final int maximumFailureRate;
+ private final Time failureInterval;
+ private final ArrayDeque<Long> failureTimestamps;
Review comment:
Why do we need a `Deque`? Wouldn't a `Queue` be enough?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManagerTest.java
##########
@@ -378,15 +382,69 @@ public void testOnError() throws Exception {
}};
}
- private static class Context {
+ @Test
+ public void testWorkerCreationInterval() throws Exception {
+ Configuration configuration = new Configuration();
+ configuration.setDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE, 1);
+ configuration.setLong(ResourceManagerOptions.WORKER_CREATION_RETRY_INTERVAL,
+ WORKER_CREATION_INTERVAL.toMilliseconds());
+ new Context(configuration) {{
+ final AtomicInteger requestCount = new AtomicInteger(0);
+
+ final List<ResourceID> tmResourceIds = new ArrayList<>();
+ tmResourceIds.add(ResourceID.generate());
+ tmResourceIds.add(ResourceID.generate());
+
+ final List<CompletableFuture<Long>> requestWorkerFromDriverFutures = new ArrayList<>();
+ requestWorkerFromDriverFutures.add(new CompletableFuture<>());
+ requestWorkerFromDriverFutures.add(new CompletableFuture<>());
+
+ driverBuilder.setRequestResourceFunction(taskExecutorProcessSpec -> {
+ int idx = requestCount.getAndIncrement();
+ assertThat(idx, lessThan(2));
+
+ requestWorkerFromDriverFutures.get(idx).complete(System.currentTimeMillis());
+ return CompletableFuture.completedFuture(tmResourceIds.get(idx));
+ });
+
+ slotManagerBuilder.setGetRequiredResourcesSupplier(() -> Collections.singletonMap(WORKER_RESOURCE_SPEC, 1));
- final Configuration flinkConfig = new Configuration();
+ runTest(() -> {
+ // received worker request, verify requesting from driver
+ CompletableFuture<Boolean> startNewWorkerFuture = runInMainThread(() ->
+ getResourceManager().startNewWorker(WORKER_RESOURCE_SPEC));
+ long t1 = requestWorkerFromDriverFutures.get(0).get(TIMEOUT_SEC, TimeUnit.SECONDS);
+ assertThat(startNewWorkerFuture.get(TIMEOUT_SEC, TimeUnit.SECONDS), is(true));
+
+ // first worker failed before register, verify requesting another worker from driver
+ runInMainThread(() -> getResourceManager().onWorkerTerminated(tmResourceIds.get(0), "terminate for testing"));
Review comment:
This only covers recording failures for workers that are failed after being successfully requested.
I think we should also cover the cases that `requestResource` future completes exceptionally.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -259,6 +286,22 @@ private void requestNewWorker(WorkerResourceSpec workerResourceSpec) {
}));
}
+ private void tryResetWorkerCreationCoolDown() {
+ if (workerCreationCoolDown.isDone()) {
+ log.info("Worker creation failed. Will not retry creating worker in {}.", workerCreationRetryInterval);
+ workerCreationCoolDown = new CompletableFuture<>();
+ getMainThreadExecutor().schedule(
Review comment:
We can use `scheduleRunAsync` directly.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -212,6 +227,9 @@ public void onPreviousAttemptWorkersRecovered(Collection<WorkerType> recoveredWo
@Override
public void onWorkerTerminated(ResourceID resourceId, String diagnostics) {
+ if (recordWorkerFailure()) {
Review comment:
We should only record failure events for workers that are not yet registered. They can be identified by checking against `currentAttemptUnregisteredWorkers`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
##########
@@ -845,6 +856,10 @@ private void registerTaskExecutorMetrics() {
resourceManagerMetricGroup.gauge(
MetricNames.NUM_REGISTERED_TASK_MANAGERS,
() -> (long) taskExecutors.size());
+
+ resourceManagerMetricGroup.meter(
Review comment:
This metric does not make any sense for standalone resource manager.
We can make `ActiveResourceManager` override this method and register this additional metric.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -109,6 +121,9 @@ public ActiveResourceManager(
this.workerNodeMap = new HashMap<>();
this.pendingWorkerCounter = new PendingWorkerCounter();
this.currentAttemptUnregisteredWorkers = new HashMap<>();
+ long retryInterval = flinkConfig.getLong(ResourceManagerOptions.WORKER_CREATION_RETRY_INTERVAL);
Review comment:
It's preferred to parse the configuration option outside `ActiveResourceManager` and pass it in as an argument, per the dependency injection principle.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRaterTest.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.ManualClock;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Test time stamp based failure rater.
+ */
+public class TimestampBasedFailureRaterTest {
+
+ @Test
+ public void testMaximumFailureCheck() {
+ TimestampBasedFailureRater rater = new TimestampBasedFailureRater(5, Time.of(10, TimeUnit.SECONDS));
+
+ for (int i = 0; i < 6; i++) {
+ rater.markEvent();
+ }
+
+ Assert.assertEquals(6.0, rater.getCurrentFailureRate(), 0.01);
+ Assert.assertTrue(rater.exceedsFailureRate());
+ }
+
+ @Test
+ public void testMovingRate() throws InterruptedException {
+ FailureRater rater = new TimestampBasedFailureRater(5, Time.of(500, TimeUnit.MILLISECONDS));
+
+ ManualClock manualClock = new ManualClock();
+ manualClock.advanceTime(System.currentTimeMillis(), TimeUnit.MILLISECONDS);
+ for (int i = 0; i < 6; i++) {
+ rater.markFailure(manualClock);
+ manualClock.advanceTime(150, TimeUnit.MILLISECONDS);
+ Thread.sleep(150);
+ }
+
+ Assert.assertEquals(3.0, rater.getCurrentFailureRate(), 0.01);
+ Assert.assertFalse(rater.exceedsFailureRate());
+ }
+}
Review comment:
I think we should also test against the interfaces in `Meter`.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRaterTest.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.ManualClock;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Test time stamp based failure rater.
+ */
+public class TimestampBasedFailureRaterTest {
Review comment:
```suggestion
public class TimestampBasedFailureRaterTest extends TestLogger {
```
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManagerTest.java
##########
@@ -378,15 +382,69 @@ public void testOnError() throws Exception {
}};
}
- private static class Context {
+ @Test
+ public void testWorkerCreationInterval() throws Exception {
+ Configuration configuration = new Configuration();
+ configuration.setDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE, 1);
+ configuration.setLong(ResourceManagerOptions.WORKER_CREATION_RETRY_INTERVAL,
+ WORKER_CREATION_INTERVAL.toMilliseconds());
+ new Context(configuration) {{
Review comment:
No need to introduce a new constructor for `Context`.
You can write into `flinkConfig` from inside `Context`, just before calling `runTest`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -246,6 +269,10 @@ private void requestNewWorker(WorkerResourceSpec workerResourceSpec) {
workerResourceSpec,
count,
exception);
+ if (recordWorkerFailure()) {
+ // if exceed failure rate try to slow down
+ tryResetWorkerCreationCoolDown();
+ }
Review comment:
Looks like `tryResetWorkerCreationCoolDown` is always called in the same if-block.
We could combine them into something like `recordWorkerFailureAndPauseWorkerCreationIfNeeded`.
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/ResourceManagerOptions.java
##########
@@ -67,6 +67,33 @@
"for streaming workloads, which may fail if there are not enough slots. Note that this configuration option does not take " +
"effect for standalone clusters, where how many slots are allocated is not controlled by Flink.");
+ /**
+ * Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting subsequent worker
+ * requests until the failure rate falls below the maximum. It is to quickly catch external dependency caused
+ * workers failure and wait for retry interval before sending new request. Be default, -1.0 is set to disable the feature.
+ */
+ public static final ConfigOption<Double> MAXIMUM_WORKERS_FAILURE_RATE = ConfigOptions
+ .key("resourcemanager.maximum-workers-failure-rate")
+ .doubleType()
+ .defaultValue(-1.0)
Review comment:
Moreover, do we really need a invalid value for disabling the feature?
In which situations would a user want to completely disable the feature? In those cases, wouldn't it be enough to set a very large rate?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 58784051e00030de475f361d314c1c0473ff2b8a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* c6cd36a8dfd854ecf63b2d248a4a28f889c873ec Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341)
* d58927642909f50571ed6242605aac564e074f89 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548939317
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricNames.java
##########
@@ -67,6 +67,7 @@ private MetricNames() {
public static final String CHECKPOINT_ALIGNMENT_TIME = "checkpointAlignmentTime";
public static final String CHECKPOINT_START_DELAY_TIME = "checkpointStartDelayNanos";
+ public static final String WORKER_FAILURE_RATE = "workFailureRate";
Review comment:
Done.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
##########
@@ -108,6 +111,8 @@
public static final String RESOURCE_MANAGER_NAME = "resourcemanager";
+ protected final Logger log = LoggerFactory.getLogger(getClass());
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548938231
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
Review comment:
Updated Accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* f59351ab8f34947c26b357ef3f87e80a85996609 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r549927391
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
Review comment:
I believe the three layers you mean are the following.
1. Meter (interface)
2. ThresholdMeter (interface)
3. TimestampBasedThresholdMeter (implementation)
My questions is, why do we need 2) and 3) separately? Do we really need the concept `TimestampBasedThresholdMeter`? What if we consider `ThresholdMeter` an implementation of `Meter` and have the following two layers?
1. Meter (interface)
2. ThresholdMeter (implementation)
I'm proposing this because:
* There's no other implementation of `ThresholdMeter` except for `TimestampBasedThresholdMeter`.
* Looking into the implementation of `TimestampBasedFailureRater`, there's nothing specifically related to *Failure*, and shouldn't the concept *Meter* already *TimestampBased*?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 58784051e00030de475f361d314c1c0473ff2b8a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279)
* c703f433eefd8591026113e4ea848c6f01cf0dd6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
}, {
"hash" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535",
"triggerID" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"triggerType" : "PUSH"
}, {
"hash" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11632",
"triggerID" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"triggerType" : "PUSH"
}, {
"hash" : "27b18b35b2826fa5fc721c3f19264a8a6cb10369",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27b18b35b2826fa5fc721c3f19264a8a6cb10369",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 8b920eb9c1f70c08f663154e15c60f6ed331394b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11632)
* 27b18b35b2826fa5fc721c3f19264a8a6cb10369 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* c6cd36a8dfd854ecf63b2d248a4a28f889c873ec Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341)
* d58927642909f50571ed6242605aac564e074f89 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547769893
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.ArrayDeque;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements FailureRater {
+ private static final int DEFAULT_TIMESTAMP_SIZE = 300;
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final int maximumFailureRate;
+ private final Time failureInterval;
+ private final ArrayDeque<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(int maximumFailureRate, Time failureInterval) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>(maximumFailureRate > 0 ? maximumFailureRate : DEFAULT_TIMESTAMP_SIZE);
Review comment:
Agree. No need to set the default size.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547752610
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/FailureRater.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.util.clock.Clock;
+
+/**
+ * Failure rate interface.
+ */
+public interface FailureRater extends Meter {
Review comment:
ThresholdMeter should be good enough, but do you see any other usage for ThresholdMeter. I would like to keep the naming here for current usage. We can change it if there is a need later. For the interface, I guess we can keep exceedsFailureRate and markFailure. As you pointed out, markEvent takes the count of occurrence as input. But we need a timestamp as parameter here for implementation and test purposes. (please check the TimestampBasedFailureRaterTest).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-752331980
@xintongsong
As we discussed offline, the TimestampBasedFailureRater is renamed to ThresholdMeter and moved to the metrics-core module. Please take a more round of review after the build is green.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-701279443
It makes sense to rebase this PR on new `ActiveResourceManager` refactoring PRs for Yarn/Mesos/k8s: #13311 and #13313 because they will be probably merged soon in 1.12. That effort changes how `ActiveResourceManager` requests workers, now using `ResourceManagerDriver`.
Also, I am not sure that the `FailureRater` is relevant for standalone deployment at all. I would consider introducing it only for `ActiveResourceManagers`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin closed pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin closed pull request #8952:
URL: https://github.com/apache/flink/pull/8952
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
}, {
"hash" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535",
"triggerID" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493)
* 50dd8154daca82c54bc3d0decc8e94b6b7cba8d6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147257697) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798)
* 58784051e00030de475f361d314c1c0473ff2b8a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
}, {
"hash" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11535",
"triggerID" : "50dd8154daca82c54bc3d0decc8e94b6b7cba8d6",
"triggerType" : "PUSH"
}, {
"hash" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11632",
"triggerID" : "8b920eb9c1f70c08f663154e15c60f6ed331394b",
"triggerType" : "PUSH"
}, {
"hash" : "27b18b35b2826fa5fc721c3f19264a8a6cb10369",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11920",
"triggerID" : "27b18b35b2826fa5fc721c3f19264a8a6cb10369",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 27b18b35b2826fa5fc721c3f19264a8a6cb10369 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11920)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547752610
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/FailureRater.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.util.clock.Clock;
+
+/**
+ * Failure rate interface.
+ */
+public interface FailureRater extends Meter {
Review comment:
ThresholdMeter should be good enough. I guess we can keep exceedsFailureRate and raname markFailure to markEvent. As you pointed out, markEvent takes the count of occurrence as input. But we need a timestamp as parameter here for implementation and test purposes, please check the TimestampBasedFailureRaterTest.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e2d99f428b8c04df9f7709f9454898e2429d3cf0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086)
* f5c053117296cab1417bfbf98a0a8e63f48def5b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-705393891
@HuangZhenQiu
I think you can reopen it (at the bottom of the comment box to the right), once you start again working on it.
Let me me know if you do not have this button `Reopen pull request` and you start to work on, I will reopen the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 7cffa19fd89147eb546988f22f2f98c911e044b4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273)
* 17290d66d3ad93815bdfea6343857b62ed182772 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548938069
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/execution/ThresholdExceedException.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.execution;
+
+/**
+ * Exception thrown when a threshold exceeds.
+ */
+public class ThresholdExceedException extends RuntimeException {
Review comment:
Agree. Moved into TimestampBasedThreshold as suggested in later comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5c053117296cab1417bfbf98a0a8e63f48def5b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-705393891
@HuangZhenQiu
I think you can reopen it (at the bottom of the comment box to the left), once you start again working on it.
Let me me know if you do not have this button `Reopen and comment` and you start to work on, I will reopen the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 17290d66d3ad93815bdfea6343857b62ed182772 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294)
* 67e9e1e4c013b78eaee9fc9d215ef400bba5cc59 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r550417892
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManager.java
##########
@@ -230,6 +271,12 @@ public void onError(Throwable exception) {
onFatalError(exception);
}
+ public static ThresholdMeter createFailureRater(Configuration configuration) {
+ double rate = configuration.getDouble(ResourceManagerOptions.MAXIMUM_WORKERS_FAILURE_RATE);
+ Preconditions.checkArgument(rate > 0, "Failure rate should be larger than 0");
+ return new ThresholdMeter(rate, Duration.ofMinutes(1));
+ }
Review comment:
It is also used in the ActiveResourceManagerTest. Probably, keeping it here will be better?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-700898600
@HuangZhenQiu
are you still interested in working on this PR? it would need a rebase on the current master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin edited a comment on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-705393891
@HuangZhenQiu
I think you can reopen it (at the bottom of the comment box to the right), once you start again working on it.
Let me me know if you do not have this button `Reopen and comment` and you start to work on, I will reopen the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r547734776
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/ResourceManagerOptions.java
##########
@@ -67,6 +67,33 @@
"for streaming workloads, which may fail if there are not enough slots. Note that this configuration option does not take " +
"effect for standalone clusters, where how many slots are allocated is not controlled by Flink.");
+ /**
+ * Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting subsequent worker
+ * requests until the failure rate falls below the maximum. It is to quickly catch external dependency caused
+ * workers failure and wait for retry interval before sending new request. Be default, -1.0 is set to disable the feature.
+ */
+ public static final ConfigOption<Double> MAXIMUM_WORKERS_FAILURE_RATE = ConfigOptions
+ .key("resourcemanager.maximum-workers-failure-rate")
+ .doubleType()
+ .defaultValue(-1.0)
+ .withDescription("Defines the maximum number of worker (YARN / Mesos) failures per minute before rejecting" +
+ " subsequent worker requests until the failure rate falls below the maximum. It is to quickly catch" +
+ " external dependency caused workers failure and terminate job accordingly." +
+ " Be default, -1.0 is set to disable the feature.");
+
+ /**
+ * Defines the worker creation interval in milliseconds. In case of worker creation failures, we should wait for an interval before
+ * trying to create new workers when the failure rate exceeds. Otherwise, ActiveResourceManager will always re-requesting
+ * the worker, which keeps the main thread busy.
+ */
+ public static final ConfigOption<Long> WORKER_CREATION_RETRY_INTERVAL = ConfigOptions
+ .key("resourcemanager.workers-creation-retry-interval")
+ .longType()
+ .defaultValue(3000L)
+ .withDescription("Defines the worker creation interval in milliseconds. In case of worker creation failures,"
+ + " we should wait for an interval before trying to create new workers when the failure rate exceeds."
+ + " Otherwise, ActiveResourceManager will always re-requesting the worker, which keeps the main thread busy.");
+
Review comment:
It makes sense as these two configurations are closely related.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548939286
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval) {
+ this(maximumFailureRate, failureInterval, System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval, Supplier<Long> customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(System.currentTimeMillis());
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() / (failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public void checkAgainstThreshold() throws ThresholdExceedException {
+ if (getCurrentFailureRate() >= maximumFailureRate) {
+ throw new ThresholdExceedException(String.format("Maximum number of failed workers %f"
+ + " is detected in Resource Manager", getCurrentFailureRate()));
+ }
+ }
+
+ private double getCurrentFailureRate() {
+ Long currentTimeStamp = System.currentTimeMillis();
Review comment:
Done.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier<Long> currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue<Long> failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval) {
+ this(maximumFailureRate, failureInterval, System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time failureInterval, Supplier<Long> customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(System.currentTimeMillis());
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() / (failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public void checkAgainstThreshold() throws ThresholdExceedException {
+ if (getCurrentFailureRate() >= maximumFailureRate) {
+ throw new ThresholdExceedException(String.format("Maximum number of failed workers %f"
+ + " is detected in Resource Manager", getCurrentFailureRate()));
+ }
+ }
+
+ private double getCurrentFailureRate() {
+ Long currentTimeStamp = System.currentTimeMillis();
+ while (!failureTimestamps.isEmpty() &&
+ currentTimeStamp - failureTimestamps.peek() > failureInterval.toMilliseconds()) {
+ failureTimestamps.remove();
+ }
+
+ return failureTimestamps.size();
Review comment:
Agree.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* adf63fa7c5c73eecf4ae0237cb3f0fde2d616082 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236)
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 99c02a09184876188e61b0149349ee0d9446660c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
}, {
"hash" : "17290d66d3ad93815bdfea6343857b62ed182772",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11294",
"triggerID" : "17290d66d3ad93815bdfea6343857b62ed182772",
"triggerType" : "PUSH"
}, {
"hash" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309",
"triggerID" : "67e9e1e4c013b78eaee9fc9d215ef400bba5cc59",
"triggerType" : "PUSH"
}, {
"hash" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341",
"triggerID" : "c6cd36a8dfd854ecf63b2d248a4a28f889c873ec",
"triggerType" : "PUSH"
}, {
"hash" : "d58927642909f50571ed6242605aac564e074f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11349",
"triggerID" : "d58927642909f50571ed6242605aac564e074f89",
"triggerType" : "PUSH"
}, {
"hash" : "70b43a088878601fa074245f88800b163641d6f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11426",
"triggerID" : "70b43a088878601fa074245f88800b163641d6f0",
"triggerType" : "PUSH"
}, {
"hash" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493",
"triggerID" : "fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* fa8c7d4a20ce5d51d831e3ccf3b48e04129e2535 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11493)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-705022743
> @HuangZhenQiu
> I am closing this PR for now. Please, reopen it if you decide to work on it again.
@azagrebin
Sorry for late reply. I will start to work on it after flink forward. Would you please reopen it for me?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-751315528
@xintongsong
Thanks for the comments. I do like the simplified version after grouping classes and utils together. For the TimestampBasedFailureRater, I would prefer to leave it as it is. Otherwise, we need to split the 100 lines of code into ThresholdMeter, TimestampBasedThresholdMeter, and FailureRater. I think we can keep it simple and refactor it once it can be usable in other components. How do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11262",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
}, {
"hash" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273",
"triggerID" : "7cffa19fd89147eb546988f22f2f98c911e044b4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 7cffa19fd89147eb546988f22f2f98c911e044b4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11273)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/119985824",
"triggerID" : "d5fa0c8c2c46bafaf2e62a02743378a1e5399b35",
"triggerType" : "PUSH"
}, {
"hash" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123621833",
"triggerID" : "cb899d83a30d8b34a4fb8ae9048bf34aeffa37f7",
"triggerType" : "PUSH"
}, {
"hash" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/123658965",
"triggerID" : "f68194461a93195faf8b90ad87e7310ef61a6460",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147151951",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "96385e84cd68896724778b80431cedb0139609e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4762",
"triggerID" : "96385e84cd68896724778b80431cedb0139609e0",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/147257697",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4798",
"triggerID" : "b59b8959f9bd54ae3a7bc4923d2d1aa5b780c331",
"triggerType" : "PUSH"
}, {
"hash" : "58784051e00030de475f361d314c1c0473ff2b8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10279",
"triggerID" : "58784051e00030de475f361d314c1c0473ff2b8a",
"triggerType" : "PUSH"
}, {
"hash" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11072",
"triggerID" : "c703f433eefd8591026113e4ea848c6f01cf0dd6",
"triggerType" : "PUSH"
}, {
"hash" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11086",
"triggerID" : "e2d99f428b8c04df9f7709f9454898e2429d3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11095",
"triggerID" : "f5c053117296cab1417bfbf98a0a8e63f48def5b",
"triggerType" : "PUSH"
}, {
"hash" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11236",
"triggerID" : "adf63fa7c5c73eecf4ae0237cb3f0fde2d616082",
"triggerType" : "PUSH"
}, {
"hash" : "d083b630115604e34b0a74498890aedbff61b2a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d083b630115604e34b0a74498890aedbff61b2a7",
"triggerType" : "PUSH"
}, {
"hash" : "99c02a09184876188e61b0149349ee0d9446660c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243",
"triggerID" : "99c02a09184876188e61b0149349ee0d9446660c",
"triggerType" : "PUSH"
}, {
"hash" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f59351ab8f34947c26b357ef3f87e80a85996609",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN
* 99c02a09184876188e61b0149349ee0d9446660c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11243)
* f59351ab8f34947c26b357ef3f87e80a85996609 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on pull request #8952: [FLINK-10868][flink-yarn] Add failure rater for resource manager
Posted by GitBox <gi...@apache.org>.
azagrebin commented on pull request #8952:
URL: https://github.com/apache/flink/pull/8952#issuecomment-700898600
@HuangZhenQiu
are you still interested in working on this? it would need a rebase on the current master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org