You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/10/06 13:08:32 UTC
[GitHub] [flink] XComp opened a new pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
XComp opened a new pull request #13547:
URL: https://github.com/apache/flink/pull/13547
## What is the purpose of the change
This change is meant to expose the managed memory usage through the REST API to make it available in the web UI.
## Brief change log
- Added new metrics for used and total managed memory
- The used managed memory is determined by accessing the MemoryManagers of all active slots of the TaskSlotTable
- Added new metrics to TaskManagerDetailsHandler
## Verifying this change
This change added tests and can be verified as follows:
- the TaskManagerDetailsHandlerTest was extended accordingly
- A test was added for retrieving all active slots in TaskSlotTableImplTest
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r506387189
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
##########
@@ -1776,6 +1776,25 @@ public ResourceID getResourceID() {
return unresolvedTaskManagerLocation.getResourceID();
}
+ public long getUsedManagedMemory() {
Review comment:
I understand what you mean saying that the `TaskExecutor` is already kind of busy doing other stuff. I'm just not sure whether it makes sense to move the logic somewhere else. It is actually a property of `TaskExecutor`. Alternatively, we could think about moving the logic into `TaskSlotTable`. Semantically, it would fit in there even more. Instead of `MetricUtils.instantiateManagedMemoryMetrics(MetricGroup, TaskExecutor)` we could use `MetricUtils.instantiateManagedMemoryMetrics(MetricGroup, TaskSlotTable)` instead. What do you think about that?
I would object moving the aggregating code into `MetricUtils`. The sole purpose of `MetricUtils` seem to be that it instantiates metrics. There is no other code. Hence, I'd like to keep it that way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
}, {
"hash" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"triggerType" : "PUSH"
}, {
"hash" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"triggerType" : "PUSH"
}, {
"hash" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945",
"triggerID" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"triggerType" : "PUSH"
}, {
"hash" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7990",
"triggerID" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"triggerType" : "PUSH"
}, {
"hash" : "b3571cf21b1d622d349f61bcb480cf18128e7a0a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8007",
"triggerID" : "b3571cf21b1d622d349f61bcb480cf18128e7a0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 5e6777513ec8c76457561937786fcba5a4d1d5af UNKNOWN
* bc9162d00efee62ab9bb8bfcb5b27e988696c717 UNKNOWN
* 7f328613d89f794e12d162ae31681311ee82bb5e Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7990)
* b3571cf21b1d622d349f61bcb480cf18128e7a0a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8007)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r505390332
##########
File path: docs/monitoring/metrics.md
##########
@@ -919,7 +919,18 @@ Thus, in order to infer the metric identifier:
<td>Mapped.TotalCapacity</td>
<td>The number of buffers in the mapped buffer pool (in bytes).</td>
<td>Gauge</td>
- </tr>
+ </tr>
+ <tr>
+ <td rowspan="2">Status.ManagedMemory</td>
+ <td>Used</td>
+ <td>The amount of managed memory currently used.</td>
+ <td>Gauge</td>
+ </tr>
+ <tr>
+ <td>Total</td>
+ <td>The total amount of managed memory.</td>
+ <td>Gauge</td>
+ </tr>
Review comment:
Good catch! I fell into the trap of mixing up metric names (`Total` vs `Max`). I guess, I'm going to discuss this matter with @zentol before fixing the docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20c70c0fec1ceb77e60e138bb334bb48f5bcadcf Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20c70c0fec1ceb77e60e138bb334bb48f5bcadcf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r508425369
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
##########
@@ -1776,6 +1776,25 @@ public ResourceID getResourceID() {
return unresolvedTaskManagerLocation.getResourceID();
}
+ public long getUsedManagedMemory() {
Review comment:
We agreed offline to move it as it would make the code cleaner. Additionally, it doesn't add more complexity to the `TaskExecutor` class as @azagrebin already mentioned.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704258348
@azagrebin Feel free to review the changes.
I am not 100% happy with the naming of the metrics (specifically, `Status.ManagedMemory.Used` vs `Status.ManagedMemory.UsedMemory` vs `Status.Flink.ManagedMemory.Used` vs ...?). But I wanted to give it a try to start a discussion on it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 562aa94b5969e0699b852626c287c5e1afc35140 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294)
* 54760df57f629890fd1de11b3f3cff80c2df74ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453)
* 73b620c0b16e24f6b985cc0361bbe35237d194d2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
}, {
"hash" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"triggerType" : "PUSH"
}, {
"hash" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"triggerType" : "PUSH"
}, {
"hash" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945",
"triggerID" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"triggerType" : "PUSH"
}, {
"hash" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7990",
"triggerID" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"triggerType" : "PUSH"
}, {
"hash" : "b3571cf21b1d622d349f61bcb480cf18128e7a0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3571cf21b1d622d349f61bcb480cf18128e7a0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 5e6777513ec8c76457561937786fcba5a4d1d5af UNKNOWN
* bc9162d00efee62ab9bb8bfcb5b27e988696c717 UNKNOWN
* 9d748ee892d2137a35a516ffc9e0a77935a1c90b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945)
* 7f328613d89f794e12d162ae31681311ee82bb5e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7990)
* b3571cf21b1d622d349f61bcb480cf18128e7a0a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
}, {
"hash" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"triggerType" : "PUSH"
}, {
"hash" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"triggerType" : "PUSH"
}, {
"hash" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945",
"triggerID" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"triggerType" : "PUSH"
}, {
"hash" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 5e6777513ec8c76457561937786fcba5a4d1d5af UNKNOWN
* bc9162d00efee62ab9bb8bfcb5b27e988696c717 UNKNOWN
* 9d748ee892d2137a35a516ffc9e0a77935a1c90b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945)
* 7f328613d89f794e12d162ae31681311ee82bb5e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r505381327
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
##########
@@ -1773,6 +1773,25 @@ public ResourceID getResourceID() {
return unresolvedTaskManagerLocation.getResourceID();
}
+ public long getUsedManagedMemory() {
+ Set<AllocationID> activeTaskAllocationIds = taskSlotTable.getActiveTaskSlotAllocationIds();
+
+ long usedMemory = 0L;
+ for (AllocationID allocationID : activeTaskAllocationIds) {
+ try {
+ usedMemory += taskSlotTable.getTaskMemoryManager(allocationID).usedMemory();
+ } catch (SlotNotFoundException e) {
+ log.debug("The task slot {} is not present anymore and will be ignore in calculating the amount of used memory.", e.getSlotID());
Review comment:
Fair enough. I reverted the `SlotNotFoundException`. I introduced them for another version of the code which is not present anymore. I considered the `SlotNotFoundException` still an improvement. But you're right: We can still do such a change if it becomes really necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
}, {
"hash" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 750fd026fc1dc7191619234901a686c6ebdf4500 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762)
* 5e6777513ec8c76457561937786fcba5a4d1d5af UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
}, {
"hash" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"triggerType" : "PUSH"
}, {
"hash" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"triggerType" : "PUSH"
}, {
"hash" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945",
"triggerID" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 750fd026fc1dc7191619234901a686c6ebdf4500 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762)
* 5e6777513ec8c76457561937786fcba5a4d1d5af UNKNOWN
* bc9162d00efee62ab9bb8bfcb5b27e988696c717 UNKNOWN
* 9d748ee892d2137a35a516ffc9e0a77935a1c90b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4239645b3f019ae4059f181c470bf9aa916f61e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274)
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 562aa94b5969e0699b852626c287c5e1afc35140 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704310910
Another thing that's up for discussion: The current implementation does provide redundant information in the REST API.
```
{
"id": "192.168.178.34:58646-64d0d8",
"path": "akka.tcp://flink@192.168.178.34:58646/user/rpc/taskmanager_0",
"dataPort": 58648,
"jmxPort": -1,
"timeSinceLastHeartbeat": 1601994279165,
"slotsNumber": 1,
"freeSlots": 0,
"totalResource": {
"cpuCores": 0,
"taskHeapMemory": 0,
"taskOffHeapMemory": 0,
"managedMemory": 0,
"networkMemory": 0,
"extendedResources": {}
},
"freeResource": {
"cpuCores": 0,
"taskHeapMemory": 0,
"taskOffHeapMemory": 0,
"managedMemory": 0,
"networkMemory": 0,
"extendedResources": {}
},
"hardware": {
"cpuCores": 12,
"physicalMemory": 17179869184,
"freeMemory": 536805376,
"managedMemory": 536870920 <-- redundant
},
"memoryConfiguration": {
"frameworkHeap": 134217728,
"taskHeap": 402653174,
"frameworkOffHeap": 134217728,
"taskOffHeap": 0,
"networkMemory": 134217730,
"managedMemory": 536870920, <-- redundant
"jvmMetaspace": 268435456,
"jvmOverhead": 1073741824,
"totalFlinkMemory": null,
"totalProcessMemory": 1811939328
},
"metrics": {
"heapUsed": 119710848,
"heapCommitted": 536805376,
"heapMax": 536805376,
"nonHeapUsed": 70984004,
"nonHeapCommitted": 455180084,
"nonHeapMax": -1,
"directCount": 4115,
"directUsed": 134750413,
"directMax": 134750412,
"mappedCount": 0,
"mappedUsed": 0,
"mappedMax": 0,
"memorySegmentsAvailable": 4092,
"memorySegmentsTotal": 4096,
"nettyShuffleMemorySegmentsAvailable": 4092,
"nettyShuffleMemorySegmentsUsed": 4,
"nettyShuffleMemorySegmentsTotal": 4096,
"nettyShuffleMemoryAvailable": 134086656,
"nettyShuffleMemoryUsed": 131072,
"nettyShuffleMemoryTotal": 134217728,
"managedMemoryAvailable": 536870920,
"managedMemoryUsed": 0,
"managedMemoryTotal": 536870920, <-- redundant
"garbageCollectors": [
{
"name": "scavenge",
"count": 2,
"time": 21
},
{
"name": "global",
"count": 0,
"time": 0
}
]
}
}
```
The example above shows that `metrics.managedMemoryTotal`, `memoryConfiguration.managedMemory`, and `hardware.managedMemory` are showing the same value. At least, `hardware.managedMemory` seems to be out of place. IMHO, we could remove it as part of this PR. Not sure about the other two, though, since it's kind of standardized.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] vthinkxie edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
vthinkxie edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-706123462
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] vthinkxie edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
vthinkxie edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-706123462
Hi, @XComp
both solutions work for me, but I suggest not to remove the metric information from the `TaskManagerDetailsHandler` since it would bring break changes to the Rest API
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4239645b3f019ae4059f181c470bf9aa916f61e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274)
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 562aa94b5969e0699b852626c287c5e1afc35140 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-706126279
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r505381976
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/SlotNotFoundException.java
##########
@@ -27,11 +27,14 @@
private static final long serialVersionUID = -883614807750137925L;
+ private final AllocationID allocationID;
+
public SlotNotFoundException(AllocationID allocationId) {
- this("Could not find slot for " + allocationId + '.');
+ super("Could not find slot for " + allocationId + '.');
+ this.allocationID = allocationId;
}
- public SlotNotFoundException(String message) {
- super(message);
+ public AllocationID getSlotID() {
Review comment:
I reverted the `SlotNotFoundException` entirely.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r505390996
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/util/MetricUtils.java
##########
@@ -132,6 +136,17 @@ public static void instantiateStatusMetrics(
instantiateCPUMetrics(jvm.addGroup("CPU"));
}
+ public static void instantiateManagedMemoryMetrics(
+ MetricGroup metricGroup,
+ TaskExecutor taskExecutor) {
+ checkNotNull(metricGroup);
+ checkNotNull(taskExecutor);
+
+ MetricGroup managedMemoryMetricGroup = metricGroup.addGroup(METRIC_GROUP_MANAGED_MEMORY);
+ managedMemoryMetricGroup.gauge(MetricNames.MEMORY_USED, taskExecutor::getUsedManagedMemory);
+ managedMemoryMetricGroup.gauge(MetricNames.MEMORY_MAX, taskExecutor::getTotalManagedMemory);
Review comment:
Yup, you might be right. I'm going to discuss the naming with @zentol.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
azagrebin commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-714292685
The CI failure looks unrelated ([FLINK-19469](https://issues.apache.org/jira/browse/FLINK-19469)), merging the PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 284588ef0ceb9223da7fcd6b72ef764cce24c2c9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420)
* 85c6be40d971fb7f4fe481e3343fb68870144add Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r506389562
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/metrics/groups/AbstractMetricGroupTest.java
##########
@@ -34,15 +34,16 @@
import org.apache.flink.runtime.metrics.ReporterSetup;
import org.apache.flink.runtime.metrics.dump.QueryScopeInfo;
import org.apache.flink.runtime.metrics.scope.ScopeFormat;
-import org.apache.flink.runtime.metrics.scope.ScopeFormats;
import org.apache.flink.runtime.metrics.util.TestReporter;
+import org.apache.flink.runtime.metrics.util.TestingMetricRegistry;
Review comment:
I will fix this when reorganizing and squashing the commits.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r501523557
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java
##########
@@ -598,6 +598,15 @@ public long availableMemory() {
return memoryBudget.getAvailableMemorySize();
}
+ /**
+ * Returns the used amount of memory handled by this memory manager.
+ *
+ * @return The used amount of memory.
+ */
+ public long usedMemory() {
+ return memoryBudget.getUsedMemorySize();
Review comment:
This could be already `getMemorySize() - availableMemory()` w/o touching `UnsafeMemoryBudget`.
Moreover, not sure about general purpose of this method, maybe we could just calculate `getMemorySize() - availableMemory()` where we query metrics.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/metrics/groups/AbstractMetricGroupTest.java
##########
@@ -34,15 +34,16 @@
import org.apache.flink.runtime.metrics.ReporterSetup;
import org.apache.flink.runtime.metrics.dump.QueryScopeInfo;
import org.apache.flink.runtime.metrics.scope.ScopeFormat;
-import org.apache.flink.runtime.metrics.scope.ScopeFormats;
import org.apache.flink.runtime.metrics.util.TestReporter;
+import org.apache.flink.runtime.metrics.util.TestingMetricRegistry;
Review comment:
I think this kind of commits should be rather called `[hotfix]`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
##########
@@ -1776,6 +1776,25 @@ public ResourceID getResourceID() {
return unresolvedTaskManagerLocation.getResourceID();
}
+ public long getUsedManagedMemory() {
Review comment:
I am not sure about the general purpose of this method.
Could we have it in `MetricUtils`?
`taskExecutorServices.getTaskSlotTable()/getManagedMemorySize ()` are already available in `TaskManagerRunner#startTaskManager` to use it in `instantiateManagedMemoryMetrics` instead of `TaskExecutor` which already has a lot of responsibilities. Then I think we should pass `TaskSlotTable` and `managedMemorySize` to `instantiateManagedMemoryMetrics` to simplify testing.
##########
File path: docs/monitoring/metrics.md
##########
@@ -919,7 +919,18 @@ Thus, in order to infer the metric identifier:
<td>Mapped.TotalCapacity</td>
<td>The number of buffers in the mapped buffer pool (in bytes).</td>
<td>Gauge</td>
- </tr>
+ </tr>
+ <tr>
+ <td rowspan="2">Status.ManagedMemory</td>
Review comment:
pity we did not call `Status.JVM.Memory` as `Status.Memory.JVM`.
The managed could be `Status.Memory.Managed`, maybe we should do it for managed like this to establish such convention. Later, JVM might be also refactored.
@zentol any more opinion about the namings?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/util/MetricUtils.java
##########
@@ -81,7 +80,7 @@ public static ProcessMetricGroup instantiateProcessMetricGroup(
final Optional<Time> systemResourceProbeInterval) {
final ProcessMetricGroup processMetricGroup = ProcessMetricGroup.create(metricRegistry, hostname);
- createAndInitializeStatusMetricGroup(processMetricGroup);
+ instantiateStatusMetricGroup(processMetricGroup);
Review comment:
also not sure the renaming is needed in a sense of preserving the git history vs minor changes, same for `getActiveTaskAllocationIdsPerJob`
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTable.java
##########
@@ -199,10 +208,11 @@ default int freeSlot(AllocationID allocationId) throws SlotNotFoundException {
/**
* Return an iterator of allocated slots for the given job id.
*
- * @param jobId for which to return the allocated slots
+ * @param jobId for which to return the allocated slots. If {@code null} is passed,
+ * all allocated slots held by this {@code TaskSlotTable} are returned.
* @return Iterator of allocated slots.
*/
- Iterator<TaskSlot<T>> getAllocatedSlots(JobID jobId);
+ Iterator<TaskSlot<T>> getAllocatedSlots(@Nullable JobID jobId);
Review comment:
I somewhat liked more the previous version which had separate methods with separate simpler responsibilities but maybe it is a bit of personal taste.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/util/MetricUtils.java
##########
@@ -81,7 +80,7 @@ public static ProcessMetricGroup instantiateProcessMetricGroup(
final Optional<Time> systemResourceProbeInterval) {
final ProcessMetricGroup processMetricGroup = ProcessMetricGroup.create(metricRegistry, hostname);
- createAndInitializeStatusMetricGroup(processMetricGroup);
+ instantiateStatusMetricGroup(processMetricGroup);
Review comment:
I think this kind of commits should be rather called `[hotfix]`.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskManagerRunnerStartupTest.java
##########
@@ -159,6 +173,66 @@ public void testStartupWhenNetworkStackFailsToInitialize() throws Exception {
}
}
+ /**
+ * Checks that all expected metrics are initialized.
+ */
+ @Test
+ public void testMetricInitialization() throws Exception {
+ Configuration cfg = createFlinkConfiguration();
+
+ List<String> registeredMetrics = new ArrayList<>();
+ startTaskManager(
+ cfg,
+ rpcService,
+ highAvailabilityServices,
+ TestingMetricRegistry.builder()
+ .setRegisterConsumer((metric, metricName, group) -> registeredMetrics.add(group.getMetricIdentifier(metricName)))
+ .setScopeFormats(ScopeFormats.fromConfig(cfg))
+ .build());
+
+ Set<String> expectedTaskManagerMetrics = Sets.newHashSet(
+ "Status.JVM.ClassLoader.ClassesLoaded",
+ "Status.JVM.ClassLoader.ClassesUnloaded",
+ "Status.JVM.GarbageCollector.scavenge.Count",
+ "Status.JVM.GarbageCollector.scavenge.Time",
+ "Status.JVM.GarbageCollector.global.Count",
+ "Status.JVM.GarbageCollector.global.Time",
+ "Status.JVM.Memory.Heap.Used",
+ "Status.JVM.Memory.Heap.Committed",
+ "Status.JVM.Memory.Heap.Max",
+ "Status.JVM.Memory.NonHeap.Used",
+ "Status.JVM.Memory.NonHeap.Committed",
+ "Status.JVM.Memory.NonHeap.Max",
+ "Status.JVM.Memory.Direct.Count",
+ "Status.JVM.Memory.Direct.MemoryUsed",
+ "Status.JVM.Memory.Direct.TotalCapacity",
+ "Status.JVM.Memory.Mapped.Count",
+ "Status.JVM.Memory.Mapped.MemoryUsed",
+ "Status.JVM.Memory.Mapped.TotalCapacity",
+ "Status.JVM.Threads.Count",
+ "Status.JVM.CPU.Load",
+ "Status.JVM.CPU.Time",
+ "Status.Network.TotalMemorySegments",
+ "Status.Network.AvailableMemorySegments",
+ "Status.Shuffle.Netty.TotalMemorySegments",
+ "Status.Shuffle.Netty.TotalMemory",
+ "Status.Shuffle.Netty.AvailableMemorySegments",
+ "Status.Shuffle.Netty.AvailableMemory",
+ "Status.Shuffle.Netty.UsedMemorySegments",
+ "Status.Shuffle.Netty.UsedMemory",
+ "Status.ManagedMemory.Used",
+ "Status.ManagedMemory.Max"
+ );
+
+ assertThat(registeredMetrics.size(), is(expectedTaskManagerMetrics.size()));
+ registeredMetrics.forEach(metric -> {
+ assertThat(metric, startsWith(".taskmanager."));
+ String metricSuffix = metric.replaceAll("\\.taskmanager\\.[^.]+\\.", "");
Review comment:
why not to include `".taskmanager."` into `expectedTaskManagerMetrics` and do:
`assertThat(registeredMetrics, containsInAnyOrder(expectedTaskManagerMetrics));`
(`expectedTaskManagerMetrics` has to be an array then `String[]`)?
There is also `assertThat(s2, everyItem(isIn(s1)));` for checking a subset.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 54760df57f629890fd1de11b3f3cff80c2df74ba Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345)
* 284588ef0ceb9223da7fcd6b72ef764cce24c2c9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r506375917
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java
##########
@@ -598,6 +598,15 @@ public long availableMemory() {
return memoryBudget.getAvailableMemorySize();
}
+ /**
+ * Returns the used amount of memory handled by this memory manager.
+ *
+ * @return The used amount of memory.
+ */
+ public long usedMemory() {
+ return memoryBudget.getUsedMemorySize();
Review comment:
Good idea. I like it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r505387654
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTable.java
##########
@@ -199,10 +208,11 @@ default int freeSlot(AllocationID allocationId) throws SlotNotFoundException {
/**
* Return an iterator of allocated slots for the given job id.
*
- * @param jobId for which to return the allocated slots
+ * @param jobId for which to return the allocated slots. If {@code null} is passed,
+ * all allocated slots held by this {@code TaskSlotTable} are returned.
* @return Iterator of allocated slots.
*/
- Iterator<TaskSlot<T>> getAllocatedSlots(JobID jobId);
+ Iterator<TaskSlot<T>> getAllocatedSlots(@Nullable JobID jobId);
Review comment:
I added the annotation since the functionality of the `TaskSlotIterator` changed. I wanted to make it explicit that there is a semantic behind passing `null` now other than returning an empty iterator (which would be the expected behavior in my opinion). Or are you saying that the JavaDoc description is enough to stress on that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-705524069
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] vthinkxie edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
vthinkxie edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-706123462
Hi, @XComp
both solutions work for me
but I suggest not to remove the metric information from the `TaskManagerDetailsHandler` since it would bring break changes to the Rest API, what about keep consistent with the previous design?
we can refactor all the Rest API later version (maybe v2?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
}, {
"hash" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"triggerType" : "PUSH"
}, {
"hash" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 750fd026fc1dc7191619234901a686c6ebdf4500 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762)
* 5e6777513ec8c76457561937786fcba5a4d1d5af UNKNOWN
* bc9162d00efee62ab9bb8bfcb5b27e988696c717 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] vthinkxie edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
vthinkxie edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-706123462
Hi, @XComp
both solutions work for me
but I suggest not to remove the metric information from the `TaskManagerDetailsHandler` since it would bring break changes to the Rest API, what about keep consistent with the previous design?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20c70c0fec1ceb77e60e138bb334bb48f5bcadcf Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238)
* 4239645b3f019ae4059f181c470bf9aa916f61e7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
zentol commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704829711
Allow me to play devil's advocate: Why are we exposing more and more things explicitly via the REST API, instead of querying the REST metric endpoints for job-/taskmanagers?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453)
* 73b620c0b16e24f6b985cc0361bbe35237d194d2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r506393225
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTable.java
##########
@@ -199,10 +208,11 @@ default int freeSlot(AllocationID allocationId) throws SlotNotFoundException {
/**
* Return an iterator of allocated slots for the given job id.
*
- * @param jobId for which to return the allocated slots
+ * @param jobId for which to return the allocated slots. If {@code null} is passed,
+ * all allocated slots held by this {@code TaskSlotTable} are returned.
* @return Iterator of allocated slots.
*/
- Iterator<TaskSlot<T>> getAllocatedSlots(JobID jobId);
+ Iterator<TaskSlot<T>> getAllocatedSlots(@Nullable JobID jobId);
Review comment:
Can you clarify what you mean by "previous version". The method signature without `@Nullable`? Or are you referring to the `TaskSlotIterator` without returning all `AllocationID`s if `null` is passed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] vthinkxie commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
vthinkxie commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-706123462
Hi, @XComp
both solutions work for me, but I suggest not to remove the metric information from the `TaskManagerDetailsHandler` since it would bring break changes in the Rest API
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 73b620c0b16e24f6b985cc0361bbe35237d194d2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625)
* 750fd026fc1dc7191619234901a686c6ebdf4500 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 73b620c0b16e24f6b985cc0361bbe35237d194d2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625)
* 750fd026fc1dc7191619234901a686c6ebdf4500 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r509100903
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/util/MetricUtils.java
##########
@@ -131,11 +147,47 @@ public static void instantiateStatusMetrics(
instantiateClassLoaderMetrics(jvm.addGroup("ClassLoader"));
instantiateGarbageCollectorMetrics(jvm.addGroup("GarbageCollector"));
- instantiateMemoryMetrics(jvm.addGroup("Memory"));
+ instantiateMemoryMetrics(jvm.addGroup(METRIC_GROUP_MEMORY));
instantiateThreadMetrics(jvm.addGroup("Threads"));
instantiateCPUMetrics(jvm.addGroup("CPU"));
}
+ public static void instantiateFlinkMemoryMetricGroup(
+ MetricGroup parentMetricGroup,
+ TaskManagerServices taskManagerServices) {
Review comment:
nit: I think `TaskManagerServices` is too heavy to pass for managed memory metrics.
I would pass `TaskSlotTable` and managed memory size.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r508376820
##########
File path: docs/monitoring/metrics.md
##########
@@ -919,7 +919,18 @@ Thus, in order to infer the metric identifier:
<td>Mapped.TotalCapacity</td>
<td>The number of buffers in the mapped buffer pool (in bytes).</td>
<td>Gauge</td>
- </tr>
+ </tr>
+ <tr>
+ <td rowspan="2">Status.ManagedMemory</td>
Review comment:
My thinking (particularly in light of different memory models between JVM/Flink, that are ideally both exposed) was to separate them like this:
```
Status.JVM.Memory(.Heap/.Off-Heap)
Status.Flink.Memory(.Managed/...)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 284588ef0ceb9223da7fcd6b72ef764cce24c2c9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420)
* 85c6be40d971fb7f4fe481e3343fb68870144add UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4239645b3f019ae4059f181c470bf9aa916f61e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274)
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-713365394
I reorganized the commits and double-checked that all issues that were raised in this PR are addressed.
@azagrebin Can you give it another check?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin closed pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
azagrebin closed pull request #13547:
URL: https://github.com/apache/flink/pull/13547
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r508455930
##########
File path: docs/monitoring/metrics.md
##########
@@ -919,7 +919,18 @@ Thus, in order to infer the metric identifier:
<td>Mapped.TotalCapacity</td>
<td>The number of buffers in the mapped buffer pool (in bytes).</td>
<td>Gauge</td>
- </tr>
+ </tr>
+ <tr>
+ <td rowspan="2">Status.ManagedMemory</td>
Review comment:
I introduced `Status.Flink.Memory.Managed` as I consider it as the best solution. 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r505388160
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTable.java
##########
@@ -246,10 +256,11 @@ default int freeSlot(AllocationID allocationId) throws SlotNotFoundException {
/**
* Return an iterator over all tasks for a given job.
*
- * @param jobId identifying the job of the requested tasks
+ * @param jobId identifying the job of the requested tasks. If {@code null} is passed,
+ * all tasks held by this {@code TaskSlotTable} are returned.
* @return Iterator over all task for a given job
*/
- Iterator<T> getTasks(JobID jobId);
+ Iterator<T> getTasks(@Nullable JobID jobId);
Review comment:
See [my comment below](https://github.com/apache/flink/pull/13547#discussion_r505387654).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-706272371
I reverted the `TaskManagerDetailsHandler` REST API changes as discussed and created [FLINK-19559](https://issues.apache.org/jira/browse/FLINK-19559) for the cleanup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 284588ef0ceb9223da7fcd6b72ef764cce24c2c9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420)
* 85c6be40d971fb7f4fe481e3343fb68870144add Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435)
* 9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r508373270
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/util/MetricUtils.java
##########
@@ -132,6 +136,17 @@ public static void instantiateStatusMetrics(
instantiateCPUMetrics(jvm.addGroup("CPU"));
}
+ public static void instantiateManagedMemoryMetrics(
+ MetricGroup metricGroup,
+ TaskExecutor taskExecutor) {
+ checkNotNull(metricGroup);
+ checkNotNull(taskExecutor);
+
+ MetricGroup managedMemoryMetricGroup = metricGroup.addGroup(METRIC_GROUP_MANAGED_MEMORY);
+ managedMemoryMetricGroup.gauge(MetricNames.MEMORY_USED, taskExecutor::getUsedManagedMemory);
+ managedMemoryMetricGroup.gauge(MetricNames.MEMORY_MAX, taskExecutor::getTotalManagedMemory);
Review comment:
We use max for various other memory-related metrics, describing the maximum amount that _could_ be used. I think we'd create more confusion by introducing new terminology.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20c70c0fec1ceb77e60e138bb334bb48f5bcadcf UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r508297589
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTable.java
##########
@@ -199,10 +208,11 @@ default int freeSlot(AllocationID allocationId) throws SlotNotFoundException {
/**
* Return an iterator of allocated slots for the given job id.
*
- * @param jobId for which to return the allocated slots
+ * @param jobId for which to return the allocated slots. If {@code null} is passed,
+ * all allocated slots held by this {@code TaskSlotTable} are returned.
* @return Iterator of allocated slots.
*/
- Iterator<TaskSlot<T>> getAllocatedSlots(JobID jobId);
+ Iterator<TaskSlot<T>> getAllocatedSlots(@Nullable JobID jobId);
Review comment:
sorry for confusion, by "previous version", I mean where the "jobId"-methods did not have additional semantics for `jobId == null` and we had a dedicated method for that.
There I wanted to suggest having separate constructors for `TaskSlotIterator` with and without `jobId`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 73b620c0b16e24f6b985cc0361bbe35237d194d2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r504815681
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
##########
@@ -1773,6 +1773,25 @@ public ResourceID getResourceID() {
return unresolvedTaskManagerLocation.getResourceID();
}
+ public long getUsedManagedMemory() {
+ Set<AllocationID> activeTaskAllocationIds = taskSlotTable.getActiveTaskSlotAllocationIds();
+
+ long usedMemory = 0L;
+ for (AllocationID allocationID : activeTaskAllocationIds) {
+ try {
+ usedMemory += taskSlotTable.getTaskMemoryManager(allocationID).usedMemory();
+ } catch (SlotNotFoundException e) {
+ log.debug("The task slot {} is not present anymore and will be ignore in calculating the amount of used memory.", e.getSlotID());
Review comment:
```suggestion
log.debug("The task slot {} is not present anymore and will be ignored in calculating the amount of used memory.", e.getSlotID());
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/util/MetricUtils.java
##########
@@ -132,6 +136,17 @@ public static void instantiateStatusMetrics(
instantiateCPUMetrics(jvm.addGroup("CPU"));
}
+ public static void instantiateManagedMemoryMetrics(
+ MetricGroup metricGroup,
+ TaskExecutor taskExecutor) {
+ checkNotNull(metricGroup);
+ checkNotNull(taskExecutor);
+
+ MetricGroup managedMemoryMetricGroup = metricGroup.addGroup(METRIC_GROUP_MANAGED_MEMORY);
+ managedMemoryMetricGroup.gauge(MetricNames.MEMORY_USED, taskExecutor::getUsedManagedMemory);
+ managedMemoryMetricGroup.gauge(MetricNames.MEMORY_MAX, taskExecutor::getTotalManagedMemory);
Review comment:
I'm not sure if "Max" is a good word here. It implies "Maximum observed value" (in the context of the GC metrics)? Can't we use "Total"?
##########
File path: docs/monitoring/metrics.md
##########
@@ -919,7 +919,18 @@ Thus, in order to infer the metric identifier:
<td>Mapped.TotalCapacity</td>
<td>The number of buffers in the mapped buffer pool (in bytes).</td>
<td>Gauge</td>
- </tr>
+ </tr>
+ <tr>
+ <td rowspan="2">Status.ManagedMemory</td>
+ <td>Used</td>
+ <td>The amount of managed memory currently used.</td>
+ <td>Gauge</td>
+ </tr>
+ <tr>
+ <td>Total</td>
+ <td>The total amount of managed memory.</td>
+ <td>Gauge</td>
+ </tr>
Review comment:
It seems that `Status.ManagedMemory.Max` is not documented? (maybe I'm misreading something here)
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
##########
@@ -1773,6 +1773,25 @@ public ResourceID getResourceID() {
return unresolvedTaskManagerLocation.getResourceID();
}
+ public long getUsedManagedMemory() {
+ Set<AllocationID> activeTaskAllocationIds = taskSlotTable.getActiveTaskSlotAllocationIds();
+
+ long usedMemory = 0L;
+ for (AllocationID allocationID : activeTaskAllocationIds) {
+ try {
+ usedMemory += taskSlotTable.getTaskMemoryManager(allocationID).usedMemory();
+ } catch (SlotNotFoundException e) {
+ log.debug("The task slot {} is not present anymore and will be ignore in calculating the amount of used memory.", e.getSlotID());
Review comment:
I'm not sure if the changes to the `SlotNotFoundException` are needed at all. We have access to the same `allocationID` in the catch clause as well.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTable.java
##########
@@ -246,10 +256,11 @@ default int freeSlot(AllocationID allocationId) throws SlotNotFoundException {
/**
* Return an iterator over all tasks for a given job.
*
- * @param jobId identifying the job of the requested tasks
+ * @param jobId identifying the job of the requested tasks. If {@code null} is passed,
+ * all tasks held by this {@code TaskSlotTable} are returned.
* @return Iterator over all task for a given job
*/
- Iterator<T> getTasks(JobID jobId);
+ Iterator<T> getTasks(@Nullable JobID jobId);
Review comment:
Why is this change necessary?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTable.java
##########
@@ -199,10 +208,11 @@ default int freeSlot(AllocationID allocationId) throws SlotNotFoundException {
/**
* Return an iterator of allocated slots for the given job id.
*
- * @param jobId for which to return the allocated slots
+ * @param jobId for which to return the allocated slots. If {@code null} is passed,
+ * all allocated slots held by this {@code TaskSlotTable} are returned.
* @return Iterator of allocated slots.
*/
- Iterator<TaskSlot<T>> getAllocatedSlots(JobID jobId);
+ Iterator<TaskSlot<T>> getAllocatedSlots(@Nullable JobID jobId);
Review comment:
Why is this change necessary?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/SlotNotFoundException.java
##########
@@ -27,11 +27,14 @@
private static final long serialVersionUID = -883614807750137925L;
+ private final AllocationID allocationID;
+
public SlotNotFoundException(AllocationID allocationId) {
- this("Could not find slot for " + allocationId + '.');
+ super("Could not find slot for " + allocationId + '.');
+ this.allocationID = allocationId;
}
- public SlotNotFoundException(String message) {
- super(message);
+ public AllocationID getSlotID() {
Review comment:
Why isn't this called `getAllocationID` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
}, {
"hash" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"triggerType" : "PUSH"
}, {
"hash" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"triggerType" : "PUSH"
}, {
"hash" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945",
"triggerID" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"triggerType" : "PUSH"
}, {
"hash" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7990",
"triggerID" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 5e6777513ec8c76457561937786fcba5a4d1d5af UNKNOWN
* bc9162d00efee62ab9bb8bfcb5b27e988696c717 UNKNOWN
* 9d748ee892d2137a35a516ffc9e0a77935a1c90b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945)
* 7f328613d89f794e12d162ae31681311ee82bb5e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7990)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-706126279
> Hi, @XComp
> both solutions work for me
> but I suggest not to remove the metric information from the `TaskManagerDetailsHandler` since it would bring break changes to the Rest API, what about keep consistent with the previous design?
> we can refactor all the Rest API later version (maybe v2?)
Thanks for your opinion. I agree with removing the metric information shouldn't be done right away. But we might consider it when thinking about updating the REST API in general having `v2` (as you mentioned as well with). There were a few things about the REST API that we could improve. Having a ticket proposing the cleanup of the metrics in the REST endpoint's response is just for documentation purposes for now.
I would propose removing the REST API update from this PR. Instead, you would use the metric endpoint when resolving [FLINK-15328](https://issues.apache.org/jira/browse/FLINK-15328).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 54760df57f629890fd1de11b3f3cff80c2df74ba Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
}, {
"hash" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"triggerType" : "PUSH"
}, {
"hash" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"triggerType" : "PUSH"
}, {
"hash" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945",
"triggerID" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 5e6777513ec8c76457561937786fcba5a4d1d5af UNKNOWN
* bc9162d00efee62ab9bb8bfcb5b27e988696c717 UNKNOWN
* 9d748ee892d2137a35a516ffc9e0a77935a1c90b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-705524069
> Allow me to play devil's advocate: Why are we exposing more and more things explicitly via the REST API, instead of querying the REST metric endpoints for job-/taskmanagers?
Good point. @vthinkxie What do you think about using the Metric REST API endpoint in addition to the `TaskManagerDetailsHandler`'s REST endpoint? This way we wouldn't have to extend this REST API. Instead, we could create a ticket to remove the metric information from the `TaskManagerDetailsHandler` endpoint as a cleanup task.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 284588ef0ceb9223da7fcd6b72ef764cce24c2c9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-713365394
I reorganized the commits and double-checked that all issues that were raised in this PR are addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-706126279
> Hi, @XComp
> both solutions work for me, but I suggest not to remove the metric information from the `TaskManagerDetailsHandler` since it would bring break changes to the Rest API
Thanks for your opinion. I agree with removing the metric information shouldn't be done right away. But we might consider it when thinking about updating the REST API in general having `v2` (as you mentioned as well with). There were a few things about the REST API that we could improve. Having a ticket proposing the cleanup of the metrics in the REST endpoint's response is just for documentation purposes for now.
I would propose removing the REST API update from this PR. Instead, you would use the metric endpoint when resolving [FLINK-15328](https://issues.apache.org/jira/browse/FLINK-15328).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 562aa94b5969e0699b852626c287c5e1afc35140 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r506455567
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskManagerRunnerStartupTest.java
##########
@@ -159,6 +173,66 @@ public void testStartupWhenNetworkStackFailsToInitialize() throws Exception {
}
}
+ /**
+ * Checks that all expected metrics are initialized.
+ */
+ @Test
+ public void testMetricInitialization() throws Exception {
+ Configuration cfg = createFlinkConfiguration();
+
+ List<String> registeredMetrics = new ArrayList<>();
+ startTaskManager(
+ cfg,
+ rpcService,
+ highAvailabilityServices,
+ TestingMetricRegistry.builder()
+ .setRegisterConsumer((metric, metricName, group) -> registeredMetrics.add(group.getMetricIdentifier(metricName)))
+ .setScopeFormats(ScopeFormats.fromConfig(cfg))
+ .build());
+
+ Set<String> expectedTaskManagerMetrics = Sets.newHashSet(
+ "Status.JVM.ClassLoader.ClassesLoaded",
+ "Status.JVM.ClassLoader.ClassesUnloaded",
+ "Status.JVM.GarbageCollector.scavenge.Count",
+ "Status.JVM.GarbageCollector.scavenge.Time",
+ "Status.JVM.GarbageCollector.global.Count",
+ "Status.JVM.GarbageCollector.global.Time",
+ "Status.JVM.Memory.Heap.Used",
+ "Status.JVM.Memory.Heap.Committed",
+ "Status.JVM.Memory.Heap.Max",
+ "Status.JVM.Memory.NonHeap.Used",
+ "Status.JVM.Memory.NonHeap.Committed",
+ "Status.JVM.Memory.NonHeap.Max",
+ "Status.JVM.Memory.Direct.Count",
+ "Status.JVM.Memory.Direct.MemoryUsed",
+ "Status.JVM.Memory.Direct.TotalCapacity",
+ "Status.JVM.Memory.Mapped.Count",
+ "Status.JVM.Memory.Mapped.MemoryUsed",
+ "Status.JVM.Memory.Mapped.TotalCapacity",
+ "Status.JVM.Threads.Count",
+ "Status.JVM.CPU.Load",
+ "Status.JVM.CPU.Time",
+ "Status.Network.TotalMemorySegments",
+ "Status.Network.AvailableMemorySegments",
+ "Status.Shuffle.Netty.TotalMemorySegments",
+ "Status.Shuffle.Netty.TotalMemory",
+ "Status.Shuffle.Netty.AvailableMemorySegments",
+ "Status.Shuffle.Netty.AvailableMemory",
+ "Status.Shuffle.Netty.UsedMemorySegments",
+ "Status.Shuffle.Netty.UsedMemory",
+ "Status.ManagedMemory.Used",
+ "Status.ManagedMemory.Max"
+ );
+
+ assertThat(registeredMetrics.size(), is(expectedTaskManagerMetrics.size()));
+ registeredMetrics.forEach(metric -> {
+ assertThat(metric, startsWith(".taskmanager."));
+ String metricSuffix = metric.replaceAll("\\.taskmanager\\.[^.]+\\.", "");
Review comment:
There's a problem with the random TaskManager ID. The substring/regex operations are solely to work around that. But fair enough: I switched to a version using `everyItem` and `isIn` instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r508435179
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/util/MetricUtils.java
##########
@@ -132,6 +136,17 @@ public static void instantiateStatusMetrics(
instantiateCPUMetrics(jvm.addGroup("CPU"));
}
+ public static void instantiateManagedMemoryMetrics(
+ MetricGroup metricGroup,
+ TaskExecutor taskExecutor) {
+ checkNotNull(metricGroup);
+ checkNotNull(taskExecutor);
+
+ MetricGroup managedMemoryMetricGroup = metricGroup.addGroup(METRIC_GROUP_MANAGED_MEMORY);
+ managedMemoryMetricGroup.gauge(MetricNames.MEMORY_USED, taskExecutor::getUsedManagedMemory);
+ managedMemoryMetricGroup.gauge(MetricNames.MEMORY_MAX, taskExecutor::getTotalManagedMemory);
Review comment:
I discussed the issue with @zentol . We agreed on using Available/Used/Total
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] vthinkxie commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
vthinkxie commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-706123462
Hi, @XComp
both solutions work for me, but I suggest not to remove the metric information from the `TaskManagerDetailsHandler` since it would bring break changes in the Rest API
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
}, {
"hash" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"triggerType" : "PUSH"
}, {
"hash" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"triggerType" : "PUSH"
}, {
"hash" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 750fd026fc1dc7191619234901a686c6ebdf4500 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762)
* 5e6777513ec8c76457561937786fcba5a4d1d5af UNKNOWN
* bc9162d00efee62ab9bb8bfcb5b27e988696c717 UNKNOWN
* 9d748ee892d2137a35a516ffc9e0a77935a1c90b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r508455414
##########
File path: docs/monitoring/metrics.md
##########
@@ -919,7 +919,18 @@ Thus, in order to infer the metric identifier:
<td>Mapped.TotalCapacity</td>
<td>The number of buffers in the mapped buffer pool (in bytes).</td>
<td>Gauge</td>
- </tr>
+ </tr>
+ <tr>
+ <td rowspan="2">Status.ManagedMemory</td>
+ <td>Used</td>
+ <td>The amount of managed memory currently used.</td>
+ <td>Gauge</td>
+ </tr>
+ <tr>
+ <td>Total</td>
+ <td>The total amount of managed memory.</td>
+ <td>Gauge</td>
+ </tr>
Review comment:
The docs have been updated as the metric group structure was changed. We sticked to `Total` and `Used` as the key.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-706126279
> Hi, @XComp
> both solutions work for me, but I suggest not to remove the metric information from the `TaskManagerDetailsHandler` since it would bring break changes to the Rest API
Thanks for your opinion. I agree with removing the metric information shouldn't be done right away. But we might consider it when thinking about updating the REST API in general having `v2` (as you mentioned as well with). There were a few things about the REST API that we could improve. Having a ticket proposing the cleanup of the metrics in the REST endpoint's response is just for documentation purposes for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 562aa94b5969e0699b852626c287c5e1afc35140 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294)
* 54760df57f629890fd1de11b3f3cff80c2df74ba UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r506406232
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/util/MetricUtils.java
##########
@@ -81,7 +80,7 @@ public static ProcessMetricGroup instantiateProcessMetricGroup(
final Optional<Time> systemResourceProbeInterval) {
final ProcessMetricGroup processMetricGroup = ProcessMetricGroup.create(metricRegistry, hostname);
- createAndInitializeStatusMetricGroup(processMetricGroup);
+ instantiateStatusMetricGroup(processMetricGroup);
Review comment:
I get the struggle between git history spam vs minor code improvement. To be honest, that one is really only a cosmetic change. I realized recently that this is something I have to work on. I'm gonna revert it.
For `getActiveTaskAllocationIdsPerJob` change, you're referring to, I would rather stick to it. The method name is not matching the actual implementation. Here, the readability of the code improves considering that the return value does not help as a hint either since it's just returning `AllocationID`. I'm gonna fix the commit message prefix when reorganizing the commits.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r508406236
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/util/MetricUtils.java
##########
@@ -132,6 +136,17 @@ public static void instantiateStatusMetrics(
instantiateCPUMetrics(jvm.addGroup("CPU"));
}
+ public static void instantiateManagedMemoryMetrics(
+ MetricGroup metricGroup,
+ TaskExecutor taskExecutor) {
+ checkNotNull(metricGroup);
+ checkNotNull(taskExecutor);
+
+ MetricGroup managedMemoryMetricGroup = metricGroup.addGroup(METRIC_GROUP_MANAGED_MEMORY);
+ managedMemoryMetricGroup.gauge(MetricNames.MEMORY_USED, taskExecutor::getUsedManagedMemory);
+ managedMemoryMetricGroup.gauge(MetricNames.MEMORY_MAX, taskExecutor::getTotalManagedMemory);
Review comment:
I agree. Unfortunately, we have already multiple different namings (`Max` vs `Total`) in place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 85c6be40d971fb7f4fe481e3343fb68870144add Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435)
* 9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
}, {
"hash" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"triggerType" : "PUSH"
}, {
"hash" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"triggerType" : "PUSH"
}, {
"hash" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945",
"triggerID" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"triggerType" : "PUSH"
}, {
"hash" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7990",
"triggerID" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"triggerType" : "PUSH"
}, {
"hash" : "b3571cf21b1d622d349f61bcb480cf18128e7a0a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8007",
"triggerID" : "b3571cf21b1d622d349f61bcb480cf18128e7a0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 5e6777513ec8c76457561937786fcba5a4d1d5af UNKNOWN
* bc9162d00efee62ab9bb8bfcb5b27e988696c717 UNKNOWN
* b3571cf21b1d622d349f61bcb480cf18128e7a0a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8007)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #13547:
URL: https://github.com/apache/flink/pull/13547#discussion_r508377512
##########
File path: docs/monitoring/metrics.md
##########
@@ -919,7 +919,18 @@ Thus, in order to infer the metric identifier:
<td>Mapped.TotalCapacity</td>
<td>The number of buffers in the mapped buffer pool (in bytes).</td>
<td>Gauge</td>
- </tr>
+ </tr>
+ <tr>
+ <td rowspan="2">Status.ManagedMemory</td>
Review comment:
Although this only really works well for memory metrics and falls apart for others (GC/cpu)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 750fd026fc1dc7191619234901a686c6ebdf4500 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7453",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
}, {
"hash" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7625",
"triggerID" : "73b620c0b16e24f6b985cc0361bbe35237d194d2",
"triggerType" : "PUSH"
}, {
"hash" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7762",
"triggerID" : "750fd026fc1dc7191619234901a686c6ebdf4500",
"triggerType" : "PUSH"
}, {
"hash" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6777513ec8c76457561937786fcba5a4d1d5af",
"triggerType" : "PUSH"
}, {
"hash" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc9162d00efee62ab9bb8bfcb5b27e988696c717",
"triggerType" : "PUSH"
}, {
"hash" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945",
"triggerID" : "9d748ee892d2137a35a516ffc9e0a77935a1c90b",
"triggerType" : "PUSH"
}, {
"hash" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7990",
"triggerID" : "7f328613d89f794e12d162ae31681311ee82bb5e",
"triggerType" : "PUSH"
}, {
"hash" : "b3571cf21b1d622d349f61bcb480cf18128e7a0a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8007",
"triggerID" : "b3571cf21b1d622d349f61bcb480cf18128e7a0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 5e6777513ec8c76457561937786fcba5a4d1d5af UNKNOWN
* bc9162d00efee62ab9bb8bfcb5b27e988696c717 UNKNOWN
* 9d748ee892d2137a35a516ffc9e0a77935a1c90b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7945)
* 7f328613d89f794e12d162ae31681311ee82bb5e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7990)
* b3571cf21b1d622d349f61bcb480cf18128e7a0a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8007)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704258387
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 392ed7a9626c675ec9f0a69ae85030004d98aef9 (Tue Oct 06 13:11:01 UTC 2020)
✅no warnings
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13547: [FLINK-14406][runtime] Exposes managed memory usage through the REST API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13547:
URL: https://github.com/apache/flink/pull/13547#issuecomment-704265372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7238",
"triggerID" : "20c70c0fec1ceb77e60e138bb334bb48f5bcadcf",
"triggerType" : "PUSH"
}, {
"hash" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7274",
"triggerID" : "4239645b3f019ae4059f181c470bf9aa916f61e7",
"triggerType" : "PUSH"
}, {
"hash" : "268d260399e598943726839a254f004a61f5ba66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "268d260399e598943726839a254f004a61f5ba66",
"triggerType" : "PUSH"
}, {
"hash" : "562aa94b5969e0699b852626c287c5e1afc35140",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7294",
"triggerID" : "562aa94b5969e0699b852626c287c5e1afc35140",
"triggerType" : "PUSH"
}, {
"hash" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7345",
"triggerID" : "54760df57f629890fd1de11b3f3cff80c2df74ba",
"triggerType" : "PUSH"
}, {
"hash" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420",
"triggerID" : "284588ef0ceb9223da7fcd6b72ef764cce24c2c9",
"triggerType" : "PUSH"
}, {
"hash" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435",
"triggerID" : "85c6be40d971fb7f4fe481e3343fb68870144add",
"triggerType" : "PUSH"
}, {
"hash" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 268d260399e598943726839a254f004a61f5ba66 UNKNOWN
* 284588ef0ceb9223da7fcd6b72ef764cce24c2c9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7420)
* 85c6be40d971fb7f4fe481e3343fb68870144add Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7435)
* 9d9cfc05d16af43bfd6ff9a2873d852eeefa5c61 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org