You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by zentol <gi...@git.apache.org> on 2016/08/12 12:43:24 UTC
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/2363
[FLINK-4389] Expose metrics to WebFrontend
This PR exposes metrics to the Webfrontend, as proposed in [FLIP-7](https://cwiki.apache.org/confluence/display/FLINK/FLIP-7%3A+Expose+metrics+to+WebInterface).
This PR builds on-top of #2300, meaning that 2866f56 is not part of the PR.
I've split the implementation into 5 commits that implement
* the generation of a separate scope string for the WebInterface
* the MetricQueryService, a separate actor running on all Job-/TaskManagers whose main purpose is to create and return a dump of the metrics when queried to do so
* the MetricStore, a nested data structure used in the WebInterface to store transmitted metrics
* the MetricFetcher, which is used by the WebInterface to fetch metrics from Job-/TaskManagers
* various MetricsHandler classes, which handle REST calls requesting specific metrics
### MetricQueryService
The MetricQueryService is an actor running inside the MetricRegistry acting like an unscheduled reporter that is queried from the outside for a report. The MetricRegistry notifies it of added/removed metrics whereas the MetricFetcher sends report requests to the JM/TM which are then forwarded to the MetricQueryService, which answers directly to the MetricFetcher.
The report is one big `Object[]`, which contains for each metric
1. the type of the metric, encoded as a byte (so that we know how many values are transmitted)
2. the fully qualified metric name (based on the separate format)
3. the value(s) of the metric (turned into Strings for Gauges)
### MetricStore
The MetricStore is a relatively simple nested data-structure that contains one HashMap<String, Object> for every JM/TM/job/task. Received metrics are added to these HashMaps based on the format string. There is only a single MetricStore instance in the WebInterface.
### MetricFetcher
The MetricFetcher initiates the transfer and cleanup of metrics. It contains the MetricStore instance, which is accessed by MetricHandlers. The fetching is only done when a handler asks for it, with a minimum duration of 10 seconds between updates. As such no fetching will be done if the metrics are not accessed with REST calls.
The fetching procedure can be summed up in pseudo-code as following:
```
fetch():
askJobManagerForJobDetails()
=> retain all metrics belonging to the given jobs
askJobManagerForMetrics()
=> add received metrics to MetricStore
askJobManagerForRegisteredTaskManagers()
=> retain all metrics belonging to registered task managers
=> for each TaskManager:
askTaskManagerForMetrics()
=> add received metrics to MetricStore
```
### MetricsHandler
The MetricsHandlers deal with two requests:
* getAllAvailableMetrics - any REST request that does not have a `get` query parameter is treated as a request for all available metrics for a given JM/TM/job/task, denoted by the REST path. The reply will be a JSON array, for example: `[{"id":"metric_1"},{"id":"metric_2"}]`
* getMetricValues - the Webfrontend can request the values for several metrics by passing a comma-separated list of metric id's as the `get` query parameter. The reply will be a JSON array of id:value pairs, for example: `[{"id":"metric_1", "value":"4"}]` or an empty string if an error occurred.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 4389_metrics_exposed
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2363.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2363
----
commit ea0e4d892717f042acf26ec9653a2371d7b21028
Author: zentol <ch...@apache.org>
Date: 2016-07-27T09:25:27Z
[FLINK-4245] Expose all defined variables
commit ea1154644566f8009ccda64a0acbdde7d59ad235
Author: zentol <ch...@apache.org>
Date: 2016-08-05T11:54:37Z
Implement Query Scope
Modifies various MetricGroups to return a separate scope for the query service.
commit 3791a94529d703351dffb284ed3d5d19f1ce272c
Author: zentol <ch...@apache.org>
Date: 2016-08-05T11:49:10Z
Implement MetricQueryService
Used on the JM/TM to create a key-value representation of all metrics.
commit a0e1418decc8a3a4b53da15dc744f1702247db9f
Author: zentol <ch...@apache.org>
Date: 2016-08-05T11:48:06Z
Implement MetricStore
Data structure used in the WebInterface to store the transmitted metrics.
commit 2bab6cc32c139f5969a276e385ed5afd6c6a46ea
Author: zentol <ch...@apache.org>
Date: 2016-08-08T12:52:01Z
Implement MetricFetcher
The MetricFetcher regularly fetches metrics from the JM and all TM's.
commit de4aeaf1e0958b49531adae198345b87ccd260bd
Author: zentol <ch...@apache.org>
Date: 2016-08-05T11:48:22Z
Implement various MetricsHandler
Handlers that answers metric related queries.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75343063
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricQueryService.java ---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.metrics;
+
+import akka.actor.ActorRef;
+import akka.actor.ActorSystem;
+import akka.actor.Props;
+import akka.actor.UntypedActor;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The MetricQueryService creates a key-value representation of all metrics currently registered with Flink when queried.
+ *
+ * It is realized as an actor and receives the following messages:
+ * - {@code Tuple3<String, Metric, AbstractMetricGroup> => Notification of added Metric}
+ * - {@code Metric => Notification of removed Metric}
+ * - {@code ActorRef => Query for metric dump}
+ */
+public class MetricQueryService extends UntypedActor {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricQueryService.class);
+
+ public static final byte CATEGORY_COUNTER = 0;
+ public static final byte CATEGORY_GAUGE = 1;
+ public static final byte CATEGORY_HISTOGRAM = 2;
+
+ private final Map<Gauge<?>, String> gauges = new HashMap<>();
+ private final Map<Counter, String> counters = new HashMap<>();
+ private final Map<Histogram, String> histograms = new HashMap<>();
+
+ private final List<Metric> toRemove = new ArrayList<>();
+
+ @Override
+ public void onReceive(Object message) throws Exception {
+ try {
+ if (message instanceof Tuple3) { // add metric
+ Tuple3<String, Metric, AbstractMetricGroup> tuple = (Tuple3<String, Metric, AbstractMetricGroup>) message;
+
+ String metricName = tuple.f0;
+ Metric metric = tuple.f1;
+ AbstractMetricGroup group = tuple.f2;
+
+ String name = group.getQueryServiceMetricIdentifier(metricName, new CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return input.replaceAll("[ :.,]", "_");
+ }
+ });
+
+ if (metric instanceof Counter) {
+ counters.put((Counter) metric, name);
+ } else if (metric instanceof Gauge) {
+ gauges.put((Gauge<?>) metric, name);
+ } else if (metric instanceof Histogram) {
+ histograms.put((Histogram) metric, name);
+ }
+ } else if (message instanceof Metric) { // remove metric
--- End diff --
The same applies to the remove metric message and the create dump. Both should get a proper message, e.g. `RemoveMetric` and `CreateDump`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75353538
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Nested data-structure to store metrics.
+ *
+ * This structure is not thread-safe.
+ */
+public class MetricStore {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricStore.class);
+
+ JobManagerMetricStore jobManager = new JobManagerMetricStore();
+ Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
+ Map<String, JobMetricStore> jobs = new HashMap<>();
--- End diff --
yes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/2363
I'll address the checkNotNull/comment formatting while merging, which I'm doing now. Thank you for looking over it again @tillrohrmann .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2363
Sorry, I meant that the hierarchy information is still encoded in a string and then re-parsed. Furthermore, the histogram data is sent as an object array without any information about the field orderings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75356320
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/AbstractMetricGroup.java ---
@@ -54,13 +53,19 @@
* return Counters, Gauges, etc to the code, to prevent exceptions in the monitored code.
* These metrics simply do not get reported any more, when created on a closed group.
*/
-public abstract class AbstractMetricGroup implements MetricGroup {
+public abstract class AbstractMetricGroup<A extends AbstractMetricGroup> implements MetricGroup {
--- End diff --
I will address this in #2300
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/2363
well...currently that is _still_ done. Whether it _will_ be done once this is merged is up in the air.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2363
I think this should be addressed (either way) before merging this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75477146
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/TaskManagerJobMetricGroup.java ---
@@ -61,9 +58,7 @@ public TaskManagerJobMetricGroup(
JobID jobId,
@Nullable String jobName) {
- super(registry, jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
-
- this.parent = checkNotNull(parent);
+ super(registry, checkNotNull(parent), jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
--- End diff --
I'm only avoiding redundant calls, that's all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75354845
--- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/messages/Messages.scala ---
@@ -47,4 +47,21 @@ object Messages {
* @return The Acknowledge case object instance.
*/
def getAcknowledge(): Acknowledge.type = Acknowledge
+
+ /**
+ * Signals that the receiver (JobManager/TaskManager) should transmit a dump of the
+ * registered metrics.
+ *
+ * This message may be send regularly by the WebInterface.
+ */
+ case object MetricRequest
+
+ /**
+ * Accessor for the case object instance, to simplify Java interoperability.
+ *
+ * @return The MetricRequest case object instance.
+ */
+ def getRequestMetrics(): AnyRef = {
--- End diff --
sure
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75451065
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricQueryService.java ---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.metrics;
+
+import akka.actor.ActorRef;
+import akka.actor.ActorSystem;
+import akka.actor.Props;
+import akka.actor.UntypedActor;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The MetricQueryService creates a key-value representation of all metrics currently registered with Flink when queried.
+ *
+ * It is realized as an actor and receives the following messages:
+ * - {@code Tuple3<String, Metric, AbstractMetricGroup> => Notification of added Metric}
+ * - {@code Metric => Notification of removed Metric}
+ * - {@code ActorRef => Query for metric dump}
+ */
+public class MetricQueryService extends UntypedActor {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricQueryService.class);
+
+ public static final byte CATEGORY_COUNTER = 0;
+ public static final byte CATEGORY_GAUGE = 1;
+ public static final byte CATEGORY_HISTOGRAM = 2;
+
+ private final Map<Gauge<?>, String> gauges = new HashMap<>();
+ private final Map<Counter, String> counters = new HashMap<>();
+ private final Map<Histogram, String> histograms = new HashMap<>();
+
+ private final List<Metric> toRemove = new ArrayList<>();
+
+ @Override
+ public void onReceive(Object message) throws Exception {
+ try {
+ if (message instanceof Tuple3) { // add metric
+ Tuple3<String, Metric, AbstractMetricGroup> tuple = (Tuple3<String, Metric, AbstractMetricGroup>) message;
+
+ String metricName = tuple.f0;
+ Metric metric = tuple.f1;
+ AbstractMetricGroup group = tuple.f2;
+
+ String name = group.getQueryServiceMetricIdentifier(metricName, new CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return input.replaceAll("[ :.,]", "_");
+ }
+ });
+
+ if (metric instanceof Counter) {
+ counters.put((Counter) metric, name);
+ } else if (metric instanceof Gauge) {
+ gauges.put((Gauge<?>) metric, name);
+ } else if (metric instanceof Histogram) {
+ histograms.put((Histogram) metric, name);
+ }
+ } else if (message instanceof Metric) { // remove metric
+ toRemove.add((Metric) message);
+ } else if (message instanceof ActorRef) { // create dump
+ Object[] dump = createDump();
+ ((ActorRef) message).tell(dump, getSelf());
+ removeMetrics();
+ } else {
+ LOG.warn("MetricQueryServiceActor received an invalid message. " + message.toString());
+ }
+ } catch (Exception e) {
+ LOG.warn("An exception occurred while processing a message.", e);
+ }
+ }
+
+ private Object[] createDump() {
+ Object[] metrics = new Object[(this.counters.size() + this.gauges.size()) * 3 + this.histograms.size() * 13];
+
+ int index = 0;
+
+ // dump counters
+ for (Map.Entry<Counter, String> counter : this.counters.entrySet()) {
+ metrics[index++] = CATEGORY_COUNTER;
+ metrics[index++] = counter.getValue();
+ metrics[index++] = counter.getKey().getCount();
+ }
+ // dump gauges
+ for (Map.Entry<Gauge<?>, String> gauge : this.gauges.entrySet()) {
+ metrics[index++] = CATEGORY_GAUGE;
+ metrics[index++] = gauge.getValue();
+ metrics[index++] = gauge.getKey().getValue().toString();
+ }
+ // dump histograms
+ for (Map.Entry<Histogram, String> histogram : this.histograms.entrySet()) {
+ HistogramStatistics stat = histogram.getKey().getStatistics();
+
+ metrics[index++] = CATEGORY_HISTOGRAM;
+ metrics[index++] = histogram.getValue();
+ metrics[index++] = stat.getMin();
+ metrics[index++] = stat.getMax();
+ metrics[index++] = stat.getMean();
+ metrics[index++] = stat.getQuantile(0.5);
+ metrics[index++] = stat.getStdDev();
+ metrics[index++] = stat.getQuantile(0.75);
+ metrics[index++] = stat.getQuantile(0.9);
+ metrics[index++] = stat.getQuantile(0.95);
+ metrics[index++] = stat.getQuantile(0.98);
+ metrics[index++] = stat.getQuantile(0.99);
+ metrics[index++] = stat.getQuantile(0.999);
+ }
+ return metrics;
+ }
+
+ private void removeMetrics() {
+ for (Metric metric : toRemove) {
+ if (metric instanceof Counter) {
+ this.counters.remove(metric);
+ } else if (metric instanceof Gauge) {
+ this.gauges.remove(metric);
+ } else if (metric instanceof Histogram) {
+ this.histograms.remove(metric);
+ }
+ }
+ }
+
+ /**
+ * Starts the MetricQueryService actor in the given actor system.
+ *
+ * @param actorSystem The actor system running the MetricQueryService
+ * @return actor reference to the MetricQueryService
+ */
+ public static ActorRef startMetricQueryService(ActorSystem actorSystem) {
+ return actorSystem.actorOf(Props.create(MetricQueryService.class));
--- End diff --
nvm, may have figured it out...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75354216
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/TaskManagerJobMetricGroup.java ---
@@ -61,9 +58,7 @@ public TaskManagerJobMetricGroup(
JobID jobId,
@Nullable String jobName) {
- super(registry, jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
-
- this.parent = checkNotNull(parent);
+ super(registry, checkNotNull(parent), jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
--- End diff --
`parent` is also used in `scopeFormat.formatScope`, as such i wanted the check before that call is made.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75340445
--- Diff: flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/metrics/AbstractMetricHandlerTest.java ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.apache.flink.runtime.webmonitor.JobManagerRetriever;
+import org.junit.Test;
+import scala.concurrent.ExecutionContext;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.powermock.api.mockito.PowerMockito.mock;
+
+public class AbstractMetricHandlerTest {
+ @Test
+ public void testHandleRequest() throws Exception {
--- End diff --
A short description would be helpful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75340742
--- Diff: flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/metrics/AbstractMetricHandlerTest.java ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.apache.flink.runtime.webmonitor.JobManagerRetriever;
+import org.junit.Test;
+import scala.concurrent.ExecutionContext;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.powermock.api.mockito.PowerMockito.mock;
+
+public class AbstractMetricHandlerTest {
+ @Test
+ public void testHandleRequest() throws Exception {
+ MetricFetcher fetcher = new MetricFetcher(mock(JobManagerRetriever.class), mock(ExecutionContext.class));
+ MetricStore store = fetcher.getMetricStore();
+
+ JobVertexMetricsHandler handler = new JobVertexMetricsHandler(fetcher);
+
+ store.add("0:abc.metric1", 0);
+ store.add("1:tmid:abc.metric2", 1);
+ store.add("2:jobid:abc.metric3", 2);
+ store.add("3:jobid:taskid:subindex:abc.metric4", 3);
+ store.add("4:jobid:taskid:subindex:opname:abc.metric5", 4);
+
+ Map<String, String> pathParams = new HashMap<>();
+ Map<String, String> queryParams = new HashMap<>();
+
+ pathParams.put("jobid", "jobid");
+ pathParams.put("vertexid", "taskid");
+
+ String availableList = handler.handleRequest(pathParams, queryParams, null);
+
+ assertEquals("[" +
+ "{\"id\":\"subindex.opname.abc.metric5\"}," +
+ "{\"id\":\"subindex.abc.metric4\"}" +
+ "]",
+ availableList);
+
+ queryParams.put("get", "subindex.opname.abc.metric5");
+
+ String metricValue = handler.handleRequest(pathParams, queryParams, null);
+
+ assertEquals("[" +
+ "{\"id\":\"subindex.opname.abc.metric5\",\"value\":\"4\"}" +
+ "]"
+ , metricValue
+ );
+
+ queryParams.put("get", "subindex.opname.abc.metric5,subindex.abc.metric4");
+
+ String metricValues = handler.handleRequest(pathParams, queryParams, null);
+
+ assertEquals("[" +
+ "{\"id\":\"subindex.opname.abc.metric5\",\"value\":\"4\"}," +
+ "{\"id\":\"subindex.abc.metric4\",\"value\":\"3\"}" +
+ "]",
+ metricValues
+ );
+ }
+
+ @Test
+ public void testInvalidListDoesNotFail() {
+ MetricFetcher fetcher = new MetricFetcher(mock(JobManagerRetriever.class), mock(ExecutionContext.class));
+ MetricStore store = fetcher.getMetricStore();
+
+ JobVertexMetricsHandler handler = new JobVertexMetricsHandler(fetcher);
+
+ store.add("0:abc.metric1", 0);
+ store.add("1:tmid:abc.metric2", 1);
+ store.add("2:jobid:abc.metric3", 2);
+ store.add("3:jobid:taskid:subindex:abc.metric4", 3);
+ store.add("4:jobid:taskid:subindex:opname:abc.metric5", 4);
+
+ Map<String, String> pathParams = new HashMap<>();
+ Map<String, String> queryParams = new HashMap<>();
+
+ pathParams.put("jobid", "jobid");
+ pathParams.put("vertexid", "taskid");
+
+ //-----invalid variable
+ pathParams.put("jobid", "nonexistant");
+
+ try {
+ handler.handleRequest(pathParams, queryParams, null);
--- End diff --
Maybe we could check that the return value is `""`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75356544
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/ComponentMetricGroup.java ---
@@ -33,19 +36,42 @@
* group could for example include the task attempt number (more fine grained identification), or
* exclude it (for continuity of the namespace across failure and recovery).
*/
-public abstract class ComponentMetricGroup extends AbstractMetricGroup {
+public abstract class ComponentMetricGroup<P extends AbstractMetricGroup> extends AbstractMetricGroup<P> {
--- End diff --
I'm not sure what you mean, but `JobMetricGroup` inherits from `ComponentMetricGroup`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75342927
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricQueryService.java ---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.metrics;
+
+import akka.actor.ActorRef;
+import akka.actor.ActorSystem;
+import akka.actor.Props;
+import akka.actor.UntypedActor;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The MetricQueryService creates a key-value representation of all metrics currently registered with Flink when queried.
+ *
+ * It is realized as an actor and receives the following messages:
+ * - {@code Tuple3<String, Metric, AbstractMetricGroup> => Notification of added Metric}
+ * - {@code Metric => Notification of removed Metric}
+ * - {@code ActorRef => Query for metric dump}
+ */
+public class MetricQueryService extends UntypedActor {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricQueryService.class);
+
+ public static final byte CATEGORY_COUNTER = 0;
+ public static final byte CATEGORY_GAUGE = 1;
+ public static final byte CATEGORY_HISTOGRAM = 2;
+
+ private final Map<Gauge<?>, String> gauges = new HashMap<>();
+ private final Map<Counter, String> counters = new HashMap<>();
+ private final Map<Histogram, String> histograms = new HashMap<>();
+
+ private final List<Metric> toRemove = new ArrayList<>();
+
+ @Override
+ public void onReceive(Object message) throws Exception {
+ try {
+ if (message instanceof Tuple3) { // add metric
--- End diff --
I think we should create proper message types instead of using a `Tuple3` here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/2363
I've updated and rebased the PR.
The scope information is now stored in a `QueryScopeInfo` inside the `MetricGroups`, for which sub-classes like `TaskQueryScopeInfo` exist. They contain fields for specific values, like the job ID, and the remaining scope not covered by these fields.
Metrics, or rather their scope, name and value(s), are now serialized with a new `MetricDumpSerializer` to a byte[] using a `Data-/ByteArrayOutputStream`.
On the other end we have the `MetricDumpDeserializer` which deserializes the metrics to a `List<MetricDump>`. A `MetricDump`is a container for the metric value, it's name the `QueryScopeInfo`. There are sub-classes for each metric type, like `CounterDump`.
Neither the `MetricQueryService` nor `MetricFetcher` know anything about the serialized format, just that it's a byte array.
There is no encoding for field orderings but tests that verify that the fields are assigned correctly. If a developer were to change the order of fields a test would fail, and the only way for this to make it into master would be if a) the test is simply changed to give a green light and b) it isn't noticed in the review, at which point all bets are of anyway. So i decided to keep it a bit simpler.
The `MetricStore#addMetric()` method has now become a bit smarter in regards to handling �Histograms`. With all values being contained in a `HistogramDump` we now only have to analyze the scope once.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r77529056
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricDump.java ---
@@ -25,6 +25,7 @@
public static final byte METRIC_CATEGORY_COUNTER = 0;
public static final byte METRIC_CATEGORY_GAUGE = 1;
public static final byte METRIC_CATEGORY_HISTOGRAM = 2;
+ public static final byte METRIC_CATEGORY_METER = 3;
--- End diff --
Sorry just saw your latest commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75346843
--- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala ---
@@ -1024,6 +1024,17 @@ class JobManager(
case RequestWebMonitorPort =>
sender() ! ResponseWebMonitorPort(webMonitorPort)
+
+ case MetricRequest =>
+ metricsRegistry match {
+ case Some(registry) =>
+ registry.getQueryService match {
+ case Some(queryService) =>
+ queryService ! sender()
+ case None =>
+ }
+ case None =>
--- End diff --
In both `None` cases you should return a failure message to the sender so that it does not have to wait until the future times out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75332220
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/AbstractMetricsHandler.java ---
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.runtime.instance.ActorGateway;
+import org.apache.flink.runtime.webmonitor.handlers.JsonFactory;
+import org.apache.flink.runtime.webmonitor.handlers.RequestHandler;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Map;
+
+/**
+ * Abstract request handler that returns a list of all available metrics or the values for a set of metrics.
+ *
+ * If the query parameters do not contain a "get" parameter the list of all metrics is returned.
+ * {@code {"available": [ { "name" : "X", "id" : "X" } ] } }
--- End diff --
That javadoc is a bit outdated, it should be `{@code [ { "id" : "X" ) ] }
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75337340
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricFetcher.java ---
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import akka.dispatch.OnSuccess;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.runtime.instance.ActorGateway;
+import org.apache.flink.runtime.instance.Instance;
+import org.apache.flink.runtime.messages.JobManagerMessages;
+import org.apache.flink.runtime.messages.Messages;
+import org.apache.flink.runtime.messages.webmonitor.JobDetails;
+import org.apache.flink.runtime.messages.webmonitor.MultipleJobsDetails;
+import org.apache.flink.runtime.messages.webmonitor.RequestJobDetails;
+import org.apache.flink.runtime.webmonitor.JobManagerRetriever;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.concurrent.ExecutionContext;
+import scala.concurrent.duration.Duration;
+import scala.concurrent.duration.FiniteDuration;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * The MetricFetcher can be used to fetch metrics from the JobManager and all registered TaskManagers.
+ *
+ * Metrics will only be fetched when {@link MetricFetcher#update()} is called, provided that a sufficient time since
+ * the last call has passed.
+ */
+public class MetricFetcher {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricFetcher.class);
+
+ private final JobManagerRetriever retriever;
+ private final ExecutionContext ctx;
+ private final FiniteDuration timeout = new FiniteDuration(Duration.create(ConfigConstants.DEFAULT_AKKA_ASK_TIMEOUT).toMillis(), TimeUnit.MILLISECONDS);
+
+ private MetricStore metrics = new MetricStore();
+
+ private long lastUpdateTime;
+
+ public MetricFetcher(JobManagerRetriever retriever, ExecutionContext ctx) {
+ this.retriever = Preconditions.checkNotNull(retriever);
+ this.ctx = Preconditions.checkNotNull(ctx);
+ }
+
+ /**
+ * Returns the MetricStore containing all stored metrics.
+ *
+ * @return MetricStore containing all stored metrics;
+ */
+ public MetricStore getMetricStore() {
+ return metrics;
+ }
+
+ /**
+ * This method can be used to signal this MetricFetcher that the metrics are still in use and should be updated.
+ */
+ public void update() {
+ synchronized (this) {
+ long currentTime = System.currentTimeMillis();
+ if (currentTime - lastUpdateTime > 10000) { // 10 seconds have passed since the last update
+ lastUpdateTime = currentTime;
+ fetchMetrics();
+ }
+ }
+ }
+
+ private void fetchMetrics() {
+ try {
+ Option<scala.Tuple2<ActorGateway, Integer>> jobManagerGatewayAndWebPort = retriever.getJobManagerGatewayAndWebPort();
+ if (jobManagerGatewayAndWebPort.isDefined()) {
+ ActorGateway jobManager = jobManagerGatewayAndWebPort.get()._1();
+
+ /**
+ * Remove all metrics that belong to a job that is not running and no longer archived.
+ */
+ jobManager.ask(new RequestJobDetails(true, true), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ MultipleJobsDetails details = (MultipleJobsDetails) result;
+ ArrayList<String> toRetain = new ArrayList<>();
+ for (JobDetails job : details.getRunningJobs()) {
+ toRetain.add(job.getJobId().toString());
+ }
+ for (JobDetails job : details.getFinishedJobs()) {
+ toRetain.add(job.getJobId().toString());
+ }
+ synchronized (metrics) {
+ metrics.jobs.keySet().retainAll(toRetain);
+ }
+ }
+ }, ctx);
+
+ /**
+ * Requests the metric dump from the job manager.
+ */
+ jobManager.ask(Messages.getRequestMetrics(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ addMetrics(result);
+ }
+ }, ctx);
+
+ /**
+ * We first request the list of all registered task managers from the job manager, and then
+ * request the respective metric dump from each task manager.
+ *
+ * All stored metrics that do not belong to a registered task manager will be removed.
+ */
+ jobManager.ask(JobManagerMessages.getRequestRegisteredTaskManagers(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ Iterable<Instance> taskManagers = ((JobManagerMessages.RegisteredTaskManagers) result).asJavaIterable();
+ List<String> activeTaskManagers = new ArrayList<>();
+ for (Instance taskManager : taskManagers) {
+ activeTaskManagers.add(taskManager.getId().toString());
+ taskManager.getActorGateway().ask(Messages.getRequestMetrics(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ addMetrics(result);
+ }
+ }, ctx);
+ }
+ synchronized (metrics) { // remove all metrics belonging to unregistered task managers
+ metrics.taskManagers.keySet().retainAll(activeTaskManagers);
+ }
+ }
+ }, ctx);
+ }
+ } catch (Exception e) {
+ LOG.warn("Exception while fetching metrics.", e);
+ }
+ }
+
+ private void addMetrics(Object result) {
+ Object[] data = (Object[]) result;
+ for (int x = 0; x < data.length; ) {
+ synchronized (metrics) {
+ switch ((byte) data[x++]) {
+ case 0:
+ case 1:
--- End diff --
Could we introduce some enums or so for these values. Maybe we could use the byte value as the ordinal of the corresponding enum value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75353675
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Nested data-structure to store metrics.
+ *
+ * This structure is not thread-safe.
+ */
+public class MetricStore {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricStore.class);
+
+ JobManagerMetricStore jobManager = new JobManagerMetricStore();
+ Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
+ Map<String, JobMetricStore> jobs = new HashMap<>();
+
+ /**
+ * Adds a metric to this MetricStore.
+ *
+ * @param name the metric identifier
+ * @param value the metric value
+ */
+ public void add(String name, Object value) {
+ TaskManagerMetricStore tm;
+ JobMetricStore job;
+ TaskMetricStore task;
+
+ try {
+ String[] components = name.split(":");
--- End diff --
yes, the MetricQueryService does that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75450989
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricQueryService.java ---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.metrics;
+
+import akka.actor.ActorRef;
+import akka.actor.ActorSystem;
+import akka.actor.Props;
+import akka.actor.UntypedActor;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The MetricQueryService creates a key-value representation of all metrics currently registered with Flink when queried.
+ *
+ * It is realized as an actor and receives the following messages:
+ * - {@code Tuple3<String, Metric, AbstractMetricGroup> => Notification of added Metric}
+ * - {@code Metric => Notification of removed Metric}
+ * - {@code ActorRef => Query for metric dump}
+ */
+public class MetricQueryService extends UntypedActor {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricQueryService.class);
+
+ public static final byte CATEGORY_COUNTER = 0;
+ public static final byte CATEGORY_GAUGE = 1;
+ public static final byte CATEGORY_HISTOGRAM = 2;
+
+ private final Map<Gauge<?>, String> gauges = new HashMap<>();
+ private final Map<Counter, String> counters = new HashMap<>();
+ private final Map<Histogram, String> histograms = new HashMap<>();
+
+ private final List<Metric> toRemove = new ArrayList<>();
+
+ @Override
+ public void onReceive(Object message) throws Exception {
+ try {
+ if (message instanceof Tuple3) { // add metric
+ Tuple3<String, Metric, AbstractMetricGroup> tuple = (Tuple3<String, Metric, AbstractMetricGroup>) message;
+
+ String metricName = tuple.f0;
+ Metric metric = tuple.f1;
+ AbstractMetricGroup group = tuple.f2;
+
+ String name = group.getQueryServiceMetricIdentifier(metricName, new CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return input.replaceAll("[ :.,]", "_");
+ }
+ });
+
+ if (metric instanceof Counter) {
+ counters.put((Counter) metric, name);
+ } else if (metric instanceof Gauge) {
+ gauges.put((Gauge<?>) metric, name);
+ } else if (metric instanceof Histogram) {
+ histograms.put((Histogram) metric, name);
+ }
+ } else if (message instanceof Metric) { // remove metric
+ toRemove.add((Metric) message);
+ } else if (message instanceof ActorRef) { // create dump
+ Object[] dump = createDump();
+ ((ActorRef) message).tell(dump, getSelf());
+ removeMetrics();
+ } else {
+ LOG.warn("MetricQueryServiceActor received an invalid message. " + message.toString());
+ }
+ } catch (Exception e) {
+ LOG.warn("An exception occurred while processing a message.", e);
+ }
+ }
+
+ private Object[] createDump() {
+ Object[] metrics = new Object[(this.counters.size() + this.gauges.size()) * 3 + this.histograms.size() * 13];
+
+ int index = 0;
+
+ // dump counters
+ for (Map.Entry<Counter, String> counter : this.counters.entrySet()) {
+ metrics[index++] = CATEGORY_COUNTER;
+ metrics[index++] = counter.getValue();
+ metrics[index++] = counter.getKey().getCount();
+ }
+ // dump gauges
+ for (Map.Entry<Gauge<?>, String> gauge : this.gauges.entrySet()) {
+ metrics[index++] = CATEGORY_GAUGE;
+ metrics[index++] = gauge.getValue();
+ metrics[index++] = gauge.getKey().getValue().toString();
+ }
+ // dump histograms
+ for (Map.Entry<Histogram, String> histogram : this.histograms.entrySet()) {
+ HistogramStatistics stat = histogram.getKey().getStatistics();
+
+ metrics[index++] = CATEGORY_HISTOGRAM;
+ metrics[index++] = histogram.getValue();
+ metrics[index++] = stat.getMin();
+ metrics[index++] = stat.getMax();
+ metrics[index++] = stat.getMean();
+ metrics[index++] = stat.getQuantile(0.5);
+ metrics[index++] = stat.getStdDev();
+ metrics[index++] = stat.getQuantile(0.75);
+ metrics[index++] = stat.getQuantile(0.9);
+ metrics[index++] = stat.getQuantile(0.95);
+ metrics[index++] = stat.getQuantile(0.98);
+ metrics[index++] = stat.getQuantile(0.99);
+ metrics[index++] = stat.getQuantile(0.999);
+ }
+ return metrics;
+ }
+
+ private void removeMetrics() {
+ for (Metric metric : toRemove) {
+ if (metric instanceof Counter) {
+ this.counters.remove(metric);
+ } else if (metric instanceof Gauge) {
+ this.gauges.remove(metric);
+ } else if (metric instanceof Histogram) {
+ this.histograms.remove(metric);
+ }
+ }
+ }
+
+ /**
+ * Starts the MetricQueryService actor in the given actor system.
+ *
+ * @param actorSystem The actor system running the MetricQueryService
+ * @return actor reference to the MetricQueryService
+ */
+ public static ActorRef startMetricQueryService(ActorSystem actorSystem) {
+ return actorSystem.actorOf(Props.create(MetricQueryService.class));
--- End diff --
`actorSystem.actorOf(Props.create(MetricQueryService.class), "foobar")` creates an actor with the name "foobar". If your actor system is the Flink actor system, then it should be reachable via `akka.tcp://flink@xxx.yyy.zzz:port/foobar` if I'm not mistaken. But you would need an actorsystem reference in the web server to do the actor resolution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75448874
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/TaskManagerJobMetricGroup.java ---
@@ -61,9 +58,7 @@ public TaskManagerJobMetricGroup(
JobID jobId,
@Nullable String jobName) {
- super(registry, jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
-
- this.parent = checkNotNull(parent);
+ super(registry, checkNotNull(parent), jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
--- End diff --
Ah ok, I see. Looking at the `formatScope` method, shouldn't rather the method enforce the preconditions than the caller?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/2363
only Gauge values are sent as strings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2363
Thanks for your contribution @zentol. I've gone over the code and made some inline comments. My main concern/question is actually the representation of metric's type and hierarchy information. I think that encoding it in a string and then re-parsing it on the receiver side to reconstruct the information is rather fragile and error-prone especially wrt maintainability. Maybe you can give me some background why you decided to do it so.
Apart from that, I think the code contains many tests, which I really like :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75355525
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricFetcher.java ---
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import akka.dispatch.OnSuccess;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.runtime.instance.ActorGateway;
+import org.apache.flink.runtime.instance.Instance;
+import org.apache.flink.runtime.messages.JobManagerMessages;
+import org.apache.flink.runtime.messages.Messages;
+import org.apache.flink.runtime.messages.webmonitor.JobDetails;
+import org.apache.flink.runtime.messages.webmonitor.MultipleJobsDetails;
+import org.apache.flink.runtime.messages.webmonitor.RequestJobDetails;
+import org.apache.flink.runtime.webmonitor.JobManagerRetriever;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.concurrent.ExecutionContext;
+import scala.concurrent.duration.Duration;
+import scala.concurrent.duration.FiniteDuration;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * The MetricFetcher can be used to fetch metrics from the JobManager and all registered TaskManagers.
+ *
+ * Metrics will only be fetched when {@link MetricFetcher#update()} is called, provided that a sufficient time since
+ * the last call has passed.
+ */
+public class MetricFetcher {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricFetcher.class);
+
+ private final JobManagerRetriever retriever;
+ private final ExecutionContext ctx;
+ private final FiniteDuration timeout = new FiniteDuration(Duration.create(ConfigConstants.DEFAULT_AKKA_ASK_TIMEOUT).toMillis(), TimeUnit.MILLISECONDS);
+
+ private MetricStore metrics = new MetricStore();
+
+ private long lastUpdateTime;
+
+ public MetricFetcher(JobManagerRetriever retriever, ExecutionContext ctx) {
+ this.retriever = Preconditions.checkNotNull(retriever);
+ this.ctx = Preconditions.checkNotNull(ctx);
+ }
+
+ /**
+ * Returns the MetricStore containing all stored metrics.
+ *
+ * @return MetricStore containing all stored metrics;
+ */
+ public MetricStore getMetricStore() {
+ return metrics;
+ }
+
+ /**
+ * This method can be used to signal this MetricFetcher that the metrics are still in use and should be updated.
+ */
+ public void update() {
+ synchronized (this) {
+ long currentTime = System.currentTimeMillis();
+ if (currentTime - lastUpdateTime > 10000) { // 10 seconds have passed since the last update
+ lastUpdateTime = currentTime;
+ fetchMetrics();
+ }
+ }
+ }
+
+ private void fetchMetrics() {
+ try {
+ Option<scala.Tuple2<ActorGateway, Integer>> jobManagerGatewayAndWebPort = retriever.getJobManagerGatewayAndWebPort();
+ if (jobManagerGatewayAndWebPort.isDefined()) {
+ ActorGateway jobManager = jobManagerGatewayAndWebPort.get()._1();
+
+ /**
+ * Remove all metrics that belong to a job that is not running and no longer archived.
+ */
+ jobManager.ask(new RequestJobDetails(true, true), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ MultipleJobsDetails details = (MultipleJobsDetails) result;
+ ArrayList<String> toRetain = new ArrayList<>();
+ for (JobDetails job : details.getRunningJobs()) {
+ toRetain.add(job.getJobId().toString());
+ }
+ for (JobDetails job : details.getFinishedJobs()) {
+ toRetain.add(job.getJobId().toString());
+ }
+ synchronized (metrics) {
+ metrics.jobs.keySet().retainAll(toRetain);
+ }
+ }
+ }, ctx);
+
+ /**
+ * Requests the metric dump from the job manager.
+ */
+ jobManager.ask(Messages.getRequestMetrics(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ addMetrics(result);
+ }
+ }, ctx);
+
+ /**
+ * We first request the list of all registered task managers from the job manager, and then
+ * request the respective metric dump from each task manager.
+ *
+ * All stored metrics that do not belong to a registered task manager will be removed.
+ */
+ jobManager.ask(JobManagerMessages.getRequestRegisteredTaskManagers(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ Iterable<Instance> taskManagers = ((JobManagerMessages.RegisteredTaskManagers) result).asJavaIterable();
+ List<String> activeTaskManagers = new ArrayList<>();
+ for (Instance taskManager : taskManagers) {
+ activeTaskManagers.add(taskManager.getId().toString());
+ taskManager.getActorGateway().ask(Messages.getRequestMetrics(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ addMetrics(result);
+ }
+ }, ctx);
+ }
+ synchronized (metrics) { // remove all metrics belonging to unregistered task managers
+ metrics.taskManagers.keySet().retainAll(activeTaskManagers);
+ }
+ }
+ }, ctx);
+ }
+ } catch (Exception e) {
+ LOG.warn("Exception while fetching metrics.", e);
+ }
+ }
+
+ private void addMetrics(Object result) {
+ Object[] data = (Object[]) result;
+ for (int x = 0; x < data.length; ) {
+ synchronized (metrics) {
+ switch ((byte) data[x++]) {
+ case 0:
+ case 1:
--- End diff --
we could use the constants defined in the MetricQueryService, or convert those to an enum.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r77528207
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricDumpSerialization.java ---
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.metrics.dump;
+
+import org.apache.commons.io.output.ByteArrayOutputStream;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.runtime.metrics.dump.QueryScopeInfo.INFO_CATEGORY_JM;
+import static org.apache.flink.runtime.metrics.dump.QueryScopeInfo.INFO_CATEGORY_JOB;
+import static org.apache.flink.runtime.metrics.dump.QueryScopeInfo.INFO_CATEGORY_OPERATOR;
+import static org.apache.flink.runtime.metrics.dump.QueryScopeInfo.INFO_CATEGORY_TASK;
+import static org.apache.flink.runtime.metrics.dump.QueryScopeInfo.INFO_CATEGORY_TM;
+
+/**
+ * Utility class for the serialization of metrics.
+ */
+public class MetricDumpSerialization {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricDumpSerialization.class);
+
+ private MetricDumpSerialization() {
+ }
+
+ // ===== Serialization =============================================================================================
+ public static class MetricDumpSerializer {
+ private ByteArrayOutputStream baos = new ByteArrayOutputStream(4096);
+ private DataOutputStream dos = new DataOutputStream(baos);
+
+ /**
+ * Serializes the given metrics and returns the resulting byte array.
+ *
+ * @param counters counters to serialize
+ * @param gauges gauges to serialize
+ * @param histograms histograms to serialize
+ * @return byte array containing the serialized metrics
+ * @throws IOException
+ */
+ public byte[] serialize(Map<Counter, Tuple2<QueryScopeInfo, String>> counters, Map<Gauge<?>, Tuple2<QueryScopeInfo, String>> gauges, Map<Histogram, Tuple2<QueryScopeInfo, String>> histograms) throws IOException {
+ baos.reset();
+ dos.writeInt(counters.size());
+ dos.writeInt(gauges.size());
+ dos.writeInt(histograms.size());
+
+ for (Map.Entry<Counter, Tuple2<QueryScopeInfo, String>> entry : counters.entrySet()) {
+ serializeMetricInfo(dos, entry.getValue().f0);
+ serializeString(dos, entry.getValue().f1);
+ serializeCounter(dos, entry.getKey());
+ }
+
+ for (Map.Entry<Gauge<?>, Tuple2<QueryScopeInfo, String>> entry : gauges.entrySet()) {
+ serializeMetricInfo(dos, entry.getValue().f0);
+ serializeString(dos, entry.getValue().f1);
+ serializeGauge(dos, entry.getKey());
+ }
+
+ for (Map.Entry<Histogram, Tuple2<QueryScopeInfo, String>> entry : histograms.entrySet()) {
+ serializeMetricInfo(dos, entry.getValue().f0);
+ serializeString(dos, entry.getValue().f1);
+ serializeHistogram(dos, entry.getKey());
+ }
+ return baos.toByteArray();
+ }
+
+ public void close() {
+ try {
+ dos.close();
+ } catch (Exception e) {
+ LOG.debug("Failed to close OutputStream.", e);
+ }
+ try {
+ baos.close();
+ } catch (Exception e) {
+ LOG.debug("Failed to close OutputStream.", e);
+ }
+ }
+ }
+
+ private static void serializeMetricInfo(DataOutputStream dos, QueryScopeInfo info) throws IOException {
+ serializeString(dos, info.scope);
+ dos.writeByte(info.getCategory());
+ switch (info.getCategory()) {
+ case INFO_CATEGORY_JM:
+ break;
+ case INFO_CATEGORY_TM:
+ String tmID = ((QueryScopeInfo.TaskManagerQueryScopeInfo) info).taskManagerID;
+ serializeString(dos, tmID);
+ break;
+ case INFO_CATEGORY_JOB:
+ QueryScopeInfo.JobQueryScopeInfo jobInfo = (QueryScopeInfo.JobQueryScopeInfo) info;
+ serializeString(dos, jobInfo.jobID);
+ break;
+ case INFO_CATEGORY_TASK:
+ QueryScopeInfo.TaskQueryScopeInfo taskInfo = (QueryScopeInfo.TaskQueryScopeInfo) info;
+ serializeString(dos, taskInfo.jobID);
+ serializeString(dos, taskInfo.vertexID);
+ dos.writeInt(taskInfo.subtaskIndex);
+ break;
+ case INFO_CATEGORY_OPERATOR:
+ QueryScopeInfo.OperatorQueryScopeInfo operatorInfo = (QueryScopeInfo.OperatorQueryScopeInfo) info;
+ serializeString(dos, operatorInfo.jobID);
+ serializeString(dos, operatorInfo.vertexID);
+ dos.writeInt(operatorInfo.subtaskIndex);
+ serializeString(dos, operatorInfo.operatorName);
+ break;
+ }
+ }
+
+ private static void serializeString(DataOutputStream dos, String string) throws IOException {
+ byte[] bytes = string.getBytes();
+ dos.writeInt(bytes.length);
+ dos.write(bytes);
+ }
+
+ private static void serializeCounter(DataOutputStream dos, Counter counter) throws IOException {
+ dos.writeLong(counter.getCount());
+ }
+
+ private static void serializeGauge(DataOutputStream dos, Gauge<?> gauge) throws IOException {
+ serializeString(dos, gauge.getValue().toString());
+ }
+
+ private static void serializeHistogram(DataOutputStream dos, Histogram histogram) throws IOException {
+ HistogramStatistics stat = histogram.getStatistics();
+
+ dos.writeLong(stat.getMin());
+ dos.writeLong(stat.getMax());
+ dos.writeDouble(stat.getMean());
+ dos.writeDouble(stat.getQuantile(0.5));
+ dos.writeDouble(stat.getStdDev());
+ dos.writeDouble(stat.getQuantile(0.75));
+ dos.writeDouble(stat.getQuantile(0.90));
+ dos.writeDouble(stat.getQuantile(0.95));
+ dos.writeDouble(stat.getQuantile(0.98));
+ dos.writeDouble(stat.getQuantile(0.99));
+ dos.writeDouble(stat.getQuantile(0.999));
+ }
+
+ // ===== Deserialization ===========================================================================================
--- End diff --
Or the dashes replaced with `=`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75343353
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java ---
@@ -144,6 +152,30 @@ public MetricRegistry(Configuration config) {
}
}
+ /**
+ * Initializes the MetricQueryService.
+ *
+ * @param actorSystem ActorSystem to create the MetricQueryService on
+ */
+ public void startQueryService(ActorSystem actorSystem) {
+ try {
+ queryService = MetricQueryService.startMetricQueryService(actorSystem);
+ } catch (Exception e) {
+ LOG.warn("Could not start MetricDumpActor. No metrics will be submitted to the WebInterface.", e);
+ }
+ }
+
+ /**
+ * Returns an ActorRef to the MetricQueryService
+ *
+ * @return ActorRef to the MetricQueryService
+ */
+ public Option<ActorRef> getQueryService() {
+ return queryService == null
--- End diff --
this is equivalent to `Option.apply(queryService)`. If queryService == null, then the apply call will return a `None`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75338668
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Nested data-structure to store metrics.
+ *
+ * This structure is not thread-safe.
+ */
+public class MetricStore {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricStore.class);
+
+ JobManagerMetricStore jobManager = new JobManagerMetricStore();
+ Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
+ Map<String, JobMetricStore> jobs = new HashMap<>();
--- End diff --
Can these fields be final?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75343567
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java ---
@@ -207,6 +239,9 @@ public void register(Metric metric, String metricName, MetricGroup group) {
}
}
}
+ if (queryService != null) {
+ notifyOfAddedMetric(queryService, metric, metricName, (AbstractMetricGroup) group);
--- End diff --
I think it's always good practice to add the class whose static method on calls in front of the static method. Otherwise you easily think that this method is somewhere defined in this class. Consequently, I would discourage static imports a little bit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75337579
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricFetcher.java ---
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import akka.dispatch.OnSuccess;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.runtime.instance.ActorGateway;
+import org.apache.flink.runtime.instance.Instance;
+import org.apache.flink.runtime.messages.JobManagerMessages;
+import org.apache.flink.runtime.messages.Messages;
+import org.apache.flink.runtime.messages.webmonitor.JobDetails;
+import org.apache.flink.runtime.messages.webmonitor.MultipleJobsDetails;
+import org.apache.flink.runtime.messages.webmonitor.RequestJobDetails;
+import org.apache.flink.runtime.webmonitor.JobManagerRetriever;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.concurrent.ExecutionContext;
+import scala.concurrent.duration.Duration;
+import scala.concurrent.duration.FiniteDuration;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * The MetricFetcher can be used to fetch metrics from the JobManager and all registered TaskManagers.
+ *
+ * Metrics will only be fetched when {@link MetricFetcher#update()} is called, provided that a sufficient time since
+ * the last call has passed.
+ */
+public class MetricFetcher {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricFetcher.class);
+
+ private final JobManagerRetriever retriever;
+ private final ExecutionContext ctx;
+ private final FiniteDuration timeout = new FiniteDuration(Duration.create(ConfigConstants.DEFAULT_AKKA_ASK_TIMEOUT).toMillis(), TimeUnit.MILLISECONDS);
+
+ private MetricStore metrics = new MetricStore();
+
+ private long lastUpdateTime;
+
+ public MetricFetcher(JobManagerRetriever retriever, ExecutionContext ctx) {
+ this.retriever = Preconditions.checkNotNull(retriever);
+ this.ctx = Preconditions.checkNotNull(ctx);
+ }
+
+ /**
+ * Returns the MetricStore containing all stored metrics.
+ *
+ * @return MetricStore containing all stored metrics;
+ */
+ public MetricStore getMetricStore() {
+ return metrics;
+ }
+
+ /**
+ * This method can be used to signal this MetricFetcher that the metrics are still in use and should be updated.
+ */
+ public void update() {
+ synchronized (this) {
+ long currentTime = System.currentTimeMillis();
+ if (currentTime - lastUpdateTime > 10000) { // 10 seconds have passed since the last update
+ lastUpdateTime = currentTime;
+ fetchMetrics();
+ }
+ }
+ }
+
+ private void fetchMetrics() {
+ try {
+ Option<scala.Tuple2<ActorGateway, Integer>> jobManagerGatewayAndWebPort = retriever.getJobManagerGatewayAndWebPort();
+ if (jobManagerGatewayAndWebPort.isDefined()) {
+ ActorGateway jobManager = jobManagerGatewayAndWebPort.get()._1();
+
+ /**
+ * Remove all metrics that belong to a job that is not running and no longer archived.
+ */
+ jobManager.ask(new RequestJobDetails(true, true), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ MultipleJobsDetails details = (MultipleJobsDetails) result;
+ ArrayList<String> toRetain = new ArrayList<>();
+ for (JobDetails job : details.getRunningJobs()) {
+ toRetain.add(job.getJobId().toString());
+ }
+ for (JobDetails job : details.getFinishedJobs()) {
+ toRetain.add(job.getJobId().toString());
+ }
+ synchronized (metrics) {
+ metrics.jobs.keySet().retainAll(toRetain);
+ }
+ }
+ }, ctx);
+
+ /**
+ * Requests the metric dump from the job manager.
+ */
+ jobManager.ask(Messages.getRequestMetrics(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ addMetrics(result);
+ }
+ }, ctx);
+
+ /**
+ * We first request the list of all registered task managers from the job manager, and then
+ * request the respective metric dump from each task manager.
+ *
+ * All stored metrics that do not belong to a registered task manager will be removed.
+ */
+ jobManager.ask(JobManagerMessages.getRequestRegisteredTaskManagers(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ Iterable<Instance> taskManagers = ((JobManagerMessages.RegisteredTaskManagers) result).asJavaIterable();
+ List<String> activeTaskManagers = new ArrayList<>();
+ for (Instance taskManager : taskManagers) {
+ activeTaskManagers.add(taskManager.getId().toString());
+ taskManager.getActorGateway().ask(Messages.getRequestMetrics(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ addMetrics(result);
+ }
+ }, ctx);
+ }
+ synchronized (metrics) { // remove all metrics belonging to unregistered task managers
+ metrics.taskManagers.keySet().retainAll(activeTaskManagers);
+ }
+ }
+ }, ctx);
+ }
+ } catch (Exception e) {
+ LOG.warn("Exception while fetching metrics.", e);
+ }
+ }
+
+ private void addMetrics(Object result) {
+ Object[] data = (Object[]) result;
+ for (int x = 0; x < data.length; ) {
+ synchronized (metrics) {
+ switch ((byte) data[x++]) {
+ case 0:
+ case 1:
+ String name = (String) data[x++];
+ Object value = data[x++];
+ metrics.add(name, value);
+ break;
+ case 2:
+ String histogramName = (String) data[x++];
+ metrics.add(histogramName + "_min", data[x++]);
+ metrics.add(histogramName + "_max", data[x++]);
+ metrics.add(histogramName + "_mean", data[x++]);
+ metrics.add(histogramName + "_median", data[x++]);
+ metrics.add(histogramName + "_stddev", data[x++]);
+ metrics.add(histogramName + "_p75", data[x++]);
+ metrics.add(histogramName + "_p90", data[x++]);
+ metrics.add(histogramName + "_p95", data[x++]);
+ metrics.add(histogramName + "_p98", data[x++]);
+ metrics.add(histogramName + "_p99", data[x++]);
+ metrics.add(histogramName + "_p999", data[x++]);
--- End diff --
This is probably quite error prone, because whenever the histogram is changed (e.g. order of serialized fields) it has to be reflected here as well. I think it would be better if we would have a serializer which does the serialization and deserialization in one place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75477986
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/TaskManagerJobMetricGroup.java ---
@@ -61,9 +58,7 @@ public TaskManagerJobMetricGroup(
JobID jobId,
@Nullable String jobName) {
- super(registry, jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
-
- this.parent = checkNotNull(parent);
+ super(registry, checkNotNull(parent), jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
--- End diff --
But then don't do it here but in the `formatScope` method. Usually the method should enforce that it's preconditions are kept.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75346431
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Nested data-structure to store metrics.
+ *
+ * This structure is not thread-safe.
+ */
+public class MetricStore {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricStore.class);
+
+ JobManagerMetricStore jobManager = new JobManagerMetricStore();
+ Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
+ Map<String, JobMetricStore> jobs = new HashMap<>();
+
+ /**
+ * Adds a metric to this MetricStore.
+ *
+ * @param name the metric identifier
+ * @param value the metric value
+ */
+ public void add(String name, Object value) {
--- End diff --
To be honest, I'm not a very big fan of encoding type and hierarchical information in a string which has to be reparsed in order to reconstruct the afore-mentioned information. The problem with this approach is that everything is very implicit and you don't have a tight coupling (in terms of format) between the sender and receiver. If something changes at the sender side, you won't notice at all that you have to change here something as well. Even at runtime the only thing you see is that you don't see the metrics. This makes it very hard to debug. I would be in favour of creating for the different types different messages, e.g. `JobManagerMetric`, `TaskManagerMetric`, `TaskMetric`, etc. These messages contain the respective information. Furthermore, for the histogram type case, you will have to do the whole parsing over and over again instead of doing it once.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75344444
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/AbstractMetricGroup.java ---
@@ -54,13 +53,19 @@
* return Counters, Gauges, etc to the code, to prevent exceptions in the monitored code.
* These metrics simply do not get reported any more, when created on a closed group.
*/
-public abstract class AbstractMetricGroup implements MetricGroup {
+public abstract class AbstractMetricGroup<A extends AbstractMetricGroup> implements MetricGroup {
--- End diff --
Type parameter description in the JavaDocs is missing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75355826
--- Diff: flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/metrics/AbstractMetricHandlerTest.java ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.apache.flink.runtime.webmonitor.JobManagerRetriever;
+import org.junit.Test;
+import scala.concurrent.ExecutionContext;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.powermock.api.mockito.PowerMockito.mock;
+
+public class AbstractMetricHandlerTest {
+ @Test
+ public void testHandleRequest() throws Exception {
+ MetricFetcher fetcher = new MetricFetcher(mock(JobManagerRetriever.class), mock(ExecutionContext.class));
+ MetricStore store = fetcher.getMetricStore();
+
+ JobVertexMetricsHandler handler = new JobVertexMetricsHandler(fetcher);
+
+ store.add("0:abc.metric1", 0);
+ store.add("1:tmid:abc.metric2", 1);
+ store.add("2:jobid:abc.metric3", 2);
+ store.add("3:jobid:taskid:subindex:abc.metric4", 3);
+ store.add("4:jobid:taskid:subindex:opname:abc.metric5", 4);
+
+ Map<String, String> pathParams = new HashMap<>();
+ Map<String, String> queryParams = new HashMap<>();
+
+ pathParams.put("jobid", "jobid");
+ pathParams.put("vertexid", "taskid");
+
+ String availableList = handler.handleRequest(pathParams, queryParams, null);
+
+ assertEquals("[" +
+ "{\"id\":\"subindex.opname.abc.metric5\"}," +
+ "{\"id\":\"subindex.abc.metric4\"}" +
+ "]",
+ availableList);
+
+ queryParams.put("get", "subindex.opname.abc.metric5");
+
+ String metricValue = handler.handleRequest(pathParams, queryParams, null);
+
+ assertEquals("[" +
+ "{\"id\":\"subindex.opname.abc.metric5\",\"value\":\"4\"}" +
+ "]"
+ , metricValue
+ );
+
+ queryParams.put("get", "subindex.opname.abc.metric5,subindex.abc.metric4");
+
+ String metricValues = handler.handleRequest(pathParams, queryParams, null);
+
+ assertEquals("[" +
+ "{\"id\":\"subindex.opname.abc.metric5\",\"value\":\"4\"}," +
+ "{\"id\":\"subindex.abc.metric4\",\"value\":\"3\"}" +
+ "]",
+ metricValues
+ );
+ }
+
+ @Test
+ public void testInvalidListDoesNotFail() {
+ MetricFetcher fetcher = new MetricFetcher(mock(JobManagerRetriever.class), mock(ExecutionContext.class));
+ MetricStore store = fetcher.getMetricStore();
+
+ JobVertexMetricsHandler handler = new JobVertexMetricsHandler(fetcher);
+
+ store.add("0:abc.metric1", 0);
+ store.add("1:tmid:abc.metric2", 1);
+ store.add("2:jobid:abc.metric3", 2);
+ store.add("3:jobid:taskid:subindex:abc.metric4", 3);
+ store.add("4:jobid:taskid:subindex:opname:abc.metric5", 4);
+
+ Map<String, String> pathParams = new HashMap<>();
+ Map<String, String> queryParams = new HashMap<>();
+
+ pathParams.put("jobid", "jobid");
+ pathParams.put("vertexid", "taskid");
+
+ //-----invalid variable
+ pathParams.put("jobid", "nonexistant");
+
+ try {
+ handler.handleRequest(pathParams, queryParams, null);
--- End diff --
good idea
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75340401
--- Diff: flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/metrics/AbstractMetricHandlerTest.java ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.apache.flink.runtime.webmonitor.JobManagerRetriever;
+import org.junit.Test;
+import scala.concurrent.ExecutionContext;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.powermock.api.mockito.PowerMockito.mock;
+
+public class AbstractMetricHandlerTest {
--- End diff --
`extends TestLogger` missing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75473604
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Nested data-structure to store metrics.
+ *
+ * This structure is not thread-safe.
+ */
+public class MetricStore {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricStore.class);
+
+ JobManagerMetricStore jobManager = new JobManagerMetricStore();
+ Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
+ Map<String, JobMetricStore> jobs = new HashMap<>();
+
+ /**
+ * Adds a metric to this MetricStore.
+ *
+ * @param name the metric identifier
+ * @param value the metric value
+ */
+ public void add(String name, Object value) {
--- End diff --
Wouldn't this just replace the object array overhead with the byte buffer allocation overhead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75474901
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/TaskManagerJobMetricGroup.java ---
@@ -61,9 +58,7 @@ public TaskManagerJobMetricGroup(
JobID jobId,
@Nullable String jobName) {
- super(registry, jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
-
- this.parent = checkNotNull(parent);
+ super(registry, checkNotNull(parent), jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
--- End diff --
Why do you do them at all if you want to save them in the first place?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2363
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75354763
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricQueryService.java ---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.metrics;
+
+import akka.actor.ActorRef;
+import akka.actor.ActorSystem;
+import akka.actor.Props;
+import akka.actor.UntypedActor;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The MetricQueryService creates a key-value representation of all metrics currently registered with Flink when queried.
+ *
+ * It is realized as an actor and receives the following messages:
+ * - {@code Tuple3<String, Metric, AbstractMetricGroup> => Notification of added Metric}
+ * - {@code Metric => Notification of removed Metric}
+ * - {@code ActorRef => Query for metric dump}
+ */
+public class MetricQueryService extends UntypedActor {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricQueryService.class);
+
+ public static final byte CATEGORY_COUNTER = 0;
+ public static final byte CATEGORY_GAUGE = 1;
+ public static final byte CATEGORY_HISTOGRAM = 2;
+
+ private final Map<Gauge<?>, String> gauges = new HashMap<>();
+ private final Map<Counter, String> counters = new HashMap<>();
+ private final Map<Histogram, String> histograms = new HashMap<>();
+
+ private final List<Metric> toRemove = new ArrayList<>();
+
+ @Override
+ public void onReceive(Object message) throws Exception {
+ try {
+ if (message instanceof Tuple3) { // add metric
+ Tuple3<String, Metric, AbstractMetricGroup> tuple = (Tuple3<String, Metric, AbstractMetricGroup>) message;
+
+ String metricName = tuple.f0;
+ Metric metric = tuple.f1;
+ AbstractMetricGroup group = tuple.f2;
+
+ String name = group.getQueryServiceMetricIdentifier(metricName, new CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return input.replaceAll("[ :.,]", "_");
+ }
+ });
+
+ if (metric instanceof Counter) {
+ counters.put((Counter) metric, name);
+ } else if (metric instanceof Gauge) {
+ gauges.put((Gauge<?>) metric, name);
+ } else if (metric instanceof Histogram) {
+ histograms.put((Histogram) metric, name);
+ }
+ } else if (message instanceof Metric) { // remove metric
+ toRemove.add((Metric) message);
+ } else if (message instanceof ActorRef) { // create dump
+ Object[] dump = createDump();
+ ((ActorRef) message).tell(dump, getSelf());
+ removeMetrics();
+ } else {
+ LOG.warn("MetricQueryServiceActor received an invalid message. " + message.toString());
+ }
+ } catch (Exception e) {
+ LOG.warn("An exception occurred while processing a message.", e);
+ }
+ }
+
+ private Object[] createDump() {
+ Object[] metrics = new Object[(this.counters.size() + this.gauges.size()) * 3 + this.histograms.size() * 13];
+
+ int index = 0;
+
+ // dump counters
+ for (Map.Entry<Counter, String> counter : this.counters.entrySet()) {
+ metrics[index++] = CATEGORY_COUNTER;
+ metrics[index++] = counter.getValue();
+ metrics[index++] = counter.getKey().getCount();
+ }
+ // dump gauges
+ for (Map.Entry<Gauge<?>, String> gauge : this.gauges.entrySet()) {
+ metrics[index++] = CATEGORY_GAUGE;
+ metrics[index++] = gauge.getValue();
+ metrics[index++] = gauge.getKey().getValue().toString();
+ }
+ // dump histograms
+ for (Map.Entry<Histogram, String> histogram : this.histograms.entrySet()) {
+ HistogramStatistics stat = histogram.getKey().getStatistics();
+
+ metrics[index++] = CATEGORY_HISTOGRAM;
+ metrics[index++] = histogram.getValue();
+ metrics[index++] = stat.getMin();
+ metrics[index++] = stat.getMax();
+ metrics[index++] = stat.getMean();
+ metrics[index++] = stat.getQuantile(0.5);
+ metrics[index++] = stat.getStdDev();
+ metrics[index++] = stat.getQuantile(0.75);
+ metrics[index++] = stat.getQuantile(0.9);
+ metrics[index++] = stat.getQuantile(0.95);
+ metrics[index++] = stat.getQuantile(0.98);
+ metrics[index++] = stat.getQuantile(0.99);
+ metrics[index++] = stat.getQuantile(0.999);
+ }
+ return metrics;
+ }
+
+ private void removeMetrics() {
+ for (Metric metric : toRemove) {
+ if (metric instanceof Counter) {
+ this.counters.remove(metric);
+ } else if (metric instanceof Gauge) {
+ this.gauges.remove(metric);
+ } else if (metric instanceof Histogram) {
+ this.histograms.remove(metric);
+ }
+ }
+ }
+
+ /**
+ * Starts the MetricQueryService actor in the given actor system.
+ *
+ * @param actorSystem The actor system running the MetricQueryService
+ * @return actor reference to the MetricQueryService
+ */
+ public static ActorRef startMetricQueryService(ActorSystem actorSystem) {
+ return actorSystem.actorOf(Props.create(MetricQueryService.class));
--- End diff --
i see, and how would i do that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75360196
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Nested data-structure to store metrics.
+ *
+ * This structure is not thread-safe.
+ */
+public class MetricStore {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricStore.class);
+
+ JobManagerMetricStore jobManager = new JobManagerMetricStore();
+ Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
+ Map<String, JobMetricStore> jobs = new HashMap<>();
+
+ /**
+ * Adds a metric to this MetricStore.
+ *
+ * @param name the metric identifier
+ * @param value the metric value
+ */
+ public void add(String name, Object value) {
--- End diff --
I think we are mixing 2 separate issues here:
A) the format of the transfer
B) how the retrieved metrics are added to the store
Your comment regarding reparsing the string for Histograms targets B. This can be fixed easily by providing specific `addHistogram/...` methods to the store.
Now, let's talk about A. Essentially, the goal was to reduce the overhead for the sender. If you look at the dumping procedure in the `MetricQueryService` you will find that for the sender the current requires the least amount of work, as far as i can tell. We don't create more elaborate containers (which btw. i would personally prefer too!) as this is another short-lived object created for every metric.
Also, in regards to sender/receiver going out of sync: we can add a test to guarantee that sender&receiver do not go out of sync.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2363
Great to hear @zentol \U0001f44d
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75451233
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/ComponentMetricGroup.java ---
@@ -33,19 +36,42 @@
* group could for example include the task attempt number (more fine grained identification), or
* exclude it (for continuity of the namespace across failure and recovery).
*/
-public abstract class ComponentMetricGroup extends AbstractMetricGroup {
+public abstract class ComponentMetricGroup<P extends AbstractMetricGroup> extends AbstractMetricGroup<P> {
--- End diff --
yes they should, the must've been removed by accident when we moved them to `flink-runtime`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75355630
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricFetcher.java ---
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import akka.dispatch.OnSuccess;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.runtime.instance.ActorGateway;
+import org.apache.flink.runtime.instance.Instance;
+import org.apache.flink.runtime.messages.JobManagerMessages;
+import org.apache.flink.runtime.messages.Messages;
+import org.apache.flink.runtime.messages.webmonitor.JobDetails;
+import org.apache.flink.runtime.messages.webmonitor.MultipleJobsDetails;
+import org.apache.flink.runtime.messages.webmonitor.RequestJobDetails;
+import org.apache.flink.runtime.webmonitor.JobManagerRetriever;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.concurrent.ExecutionContext;
+import scala.concurrent.duration.Duration;
+import scala.concurrent.duration.FiniteDuration;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * The MetricFetcher can be used to fetch metrics from the JobManager and all registered TaskManagers.
+ *
+ * Metrics will only be fetched when {@link MetricFetcher#update()} is called, provided that a sufficient time since
+ * the last call has passed.
+ */
+public class MetricFetcher {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricFetcher.class);
+
+ private final JobManagerRetriever retriever;
+ private final ExecutionContext ctx;
+ private final FiniteDuration timeout = new FiniteDuration(Duration.create(ConfigConstants.DEFAULT_AKKA_ASK_TIMEOUT).toMillis(), TimeUnit.MILLISECONDS);
+
+ private MetricStore metrics = new MetricStore();
+
+ private long lastUpdateTime;
+
+ public MetricFetcher(JobManagerRetriever retriever, ExecutionContext ctx) {
+ this.retriever = Preconditions.checkNotNull(retriever);
+ this.ctx = Preconditions.checkNotNull(ctx);
+ }
+
+ /**
+ * Returns the MetricStore containing all stored metrics.
+ *
+ * @return MetricStore containing all stored metrics;
+ */
+ public MetricStore getMetricStore() {
+ return metrics;
+ }
+
+ /**
+ * This method can be used to signal this MetricFetcher that the metrics are still in use and should be updated.
+ */
+ public void update() {
+ synchronized (this) {
+ long currentTime = System.currentTimeMillis();
+ if (currentTime - lastUpdateTime > 10000) { // 10 seconds have passed since the last update
+ lastUpdateTime = currentTime;
+ fetchMetrics();
+ }
+ }
+ }
+
+ private void fetchMetrics() {
+ try {
+ Option<scala.Tuple2<ActorGateway, Integer>> jobManagerGatewayAndWebPort = retriever.getJobManagerGatewayAndWebPort();
+ if (jobManagerGatewayAndWebPort.isDefined()) {
+ ActorGateway jobManager = jobManagerGatewayAndWebPort.get()._1();
+
+ /**
+ * Remove all metrics that belong to a job that is not running and no longer archived.
+ */
+ jobManager.ask(new RequestJobDetails(true, true), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ MultipleJobsDetails details = (MultipleJobsDetails) result;
+ ArrayList<String> toRetain = new ArrayList<>();
+ for (JobDetails job : details.getRunningJobs()) {
+ toRetain.add(job.getJobId().toString());
+ }
+ for (JobDetails job : details.getFinishedJobs()) {
+ toRetain.add(job.getJobId().toString());
+ }
+ synchronized (metrics) {
+ metrics.jobs.keySet().retainAll(toRetain);
+ }
+ }
+ }, ctx);
+
+ /**
+ * Requests the metric dump from the job manager.
+ */
+ jobManager.ask(Messages.getRequestMetrics(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ addMetrics(result);
+ }
+ }, ctx);
+
+ /**
+ * We first request the list of all registered task managers from the job manager, and then
+ * request the respective metric dump from each task manager.
+ *
+ * All stored metrics that do not belong to a registered task manager will be removed.
+ */
+ jobManager.ask(JobManagerMessages.getRequestRegisteredTaskManagers(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ Iterable<Instance> taskManagers = ((JobManagerMessages.RegisteredTaskManagers) result).asJavaIterable();
+ List<String> activeTaskManagers = new ArrayList<>();
+ for (Instance taskManager : taskManagers) {
+ activeTaskManagers.add(taskManager.getId().toString());
+ taskManager.getActorGateway().ask(Messages.getRequestMetrics(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ addMetrics(result);
+ }
+ }, ctx);
+ }
+ synchronized (metrics) { // remove all metrics belonging to unregistered task managers
+ metrics.taskManagers.keySet().retainAll(activeTaskManagers);
+ }
+ }
+ }, ctx);
+ }
+ } catch (Exception e) {
+ LOG.warn("Exception while fetching metrics.", e);
+ }
+ }
+
+ private void addMetrics(Object result) {
+ Object[] data = (Object[]) result;
+ for (int x = 0; x < data.length; ) {
+ synchronized (metrics) {
+ switch ((byte) data[x++]) {
+ case 0:
+ case 1:
+ String name = (String) data[x++];
+ Object value = data[x++];
+ metrics.add(name, value);
+ break;
+ case 2:
+ String histogramName = (String) data[x++];
+ metrics.add(histogramName + "_min", data[x++]);
+ metrics.add(histogramName + "_max", data[x++]);
+ metrics.add(histogramName + "_mean", data[x++]);
+ metrics.add(histogramName + "_median", data[x++]);
+ metrics.add(histogramName + "_stddev", data[x++]);
+ metrics.add(histogramName + "_p75", data[x++]);
+ metrics.add(histogramName + "_p90", data[x++]);
+ metrics.add(histogramName + "_p95", data[x++]);
+ metrics.add(histogramName + "_p98", data[x++]);
+ metrics.add(histogramName + "_p99", data[x++]);
+ metrics.add(histogramName + "_p999", data[x++]);
--- End diff --
sounds reasonable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r77540202
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -35,109 +46,111 @@
final Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
final Map<String, JobMetricStore> jobs = new HashMap<>();
- /**
- * Adds a metric to this MetricStore.
- *
- * @param name the metric identifier
- * @param value the metric value
- */
- public void add(String name, Object value) {
- TaskManagerMetricStore tm;
- JobMetricStore job;
- TaskMetricStore task;
-
+ public void add(MetricDump metric) {
try {
- String[] components = name.split(":");
- switch (components[0]) {
- /**
- * JobManagerMetricStore metric
- * format: 0:<user_scope>.<name>
- */
- case "0":
- jobManager.metrics.put(components[1], value);
- break;
- /**
- * TaskManager metric
- * format: 1:<tm_id>:<user_scope>.<name>
- */
- case "1":
- if (components.length != 3) {
- break;
- }
- tm = taskManagers.get(components[1]);
+ QueryScopeInfo info = metric.scopeInfo;
+ TaskManagerMetricStore tm;
+ JobMetricStore job;
+ TaskMetricStore task;
+
+ String name = info.scope.isEmpty()
+ ? metric.name
+ : info.scope + "." + metric.name;
+
+ if (name.isEmpty()) { // malformed transmission
+ return;
+ }
+
+ switch (info.getCategory()) {
+ case INFO_CATEGORY_JM:
--- End diff --
That is true. Performance-wise it is the more efficient way to execute it, no doubt. I was just wondering whether this is not a case of premature optimization with the price of harder maintainability.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r77528928
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricDump.java ---
@@ -25,6 +25,7 @@
public static final byte METRIC_CATEGORY_COUNTER = 0;
public static final byte METRIC_CATEGORY_GAUGE = 1;
public static final byte METRIC_CATEGORY_HISTOGRAM = 2;
+ public static final byte METRIC_CATEGORY_METER = 3;
--- End diff --
No test added for meter metric type.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2363
I think the changes look good. Thanks for your work @zentol :-) I only had a minor question whether we can substitute the explicit category information by the type information of the metric dumps and the `QueryScopeInfo` instances (not for serialization but in the `MetricStore`).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75344832
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/ComponentMetricGroup.java ---
@@ -33,19 +36,42 @@
* group could for example include the task attempt number (more fine grained identification), or
* exclude it (for continuity of the namespace across failure and recovery).
*/
-public abstract class ComponentMetricGroup extends AbstractMetricGroup {
+public abstract class ComponentMetricGroup<P extends AbstractMetricGroup> extends AbstractMetricGroup<P> {
--- End diff --
Is this an internal component like the `JobMetricGroup`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75448477
--- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala ---
@@ -1024,6 +1024,17 @@ class JobManager(
case RequestWebMonitorPort =>
sender() ! ResponseWebMonitorPort(webMonitorPort)
+
+ case MetricRequest =>
+ metricsRegistry match {
+ case Some(registry) =>
+ registry.getQueryService match {
+ case Some(queryService) =>
+ queryService ! sender()
+ case None =>
+ }
+ case None =>
--- End diff --
If we directly ask the query service we don't need this code anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75472729
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/TaskManagerJobMetricGroup.java ---
@@ -61,9 +58,7 @@ public TaskManagerJobMetricGroup(
JobID jobId,
@Nullable String jobName) {
- super(registry, jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
-
- this.parent = checkNotNull(parent);
+ super(registry, checkNotNull(parent), jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
--- End diff --
Generally i would agree, but this way save a checkNotNull call...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75356190
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricQueryService.java ---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.metrics;
+
+import akka.actor.ActorRef;
+import akka.actor.ActorSystem;
+import akka.actor.Props;
+import akka.actor.UntypedActor;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The MetricQueryService creates a key-value representation of all metrics currently registered with Flink when queried.
+ *
+ * It is realized as an actor and receives the following messages:
+ * - {@code Tuple3<String, Metric, AbstractMetricGroup> => Notification of added Metric}
+ * - {@code Metric => Notification of removed Metric}
+ * - {@code ActorRef => Query for metric dump}
+ */
+public class MetricQueryService extends UntypedActor {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricQueryService.class);
+
+ public static final byte CATEGORY_COUNTER = 0;
+ public static final byte CATEGORY_GAUGE = 1;
+ public static final byte CATEGORY_HISTOGRAM = 2;
+
+ private final Map<Gauge<?>, String> gauges = new HashMap<>();
+ private final Map<Counter, String> counters = new HashMap<>();
+ private final Map<Histogram, String> histograms = new HashMap<>();
+
+ private final List<Metric> toRemove = new ArrayList<>();
+
+ @Override
+ public void onReceive(Object message) throws Exception {
+ try {
+ if (message instanceof Tuple3) { // add metric
--- End diff --
agreed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75340930
--- Diff: flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/metrics/AbstractMetricHandlerTest.java ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.apache.flink.runtime.webmonitor.JobManagerRetriever;
+import org.junit.Test;
+import scala.concurrent.ExecutionContext;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.powermock.api.mockito.PowerMockito.mock;
+
+public class AbstractMetricHandlerTest {
--- End diff --
Should be added to all tests because this makes debugging with Travis logs considerably easier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75339017
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Nested data-structure to store metrics.
+ *
+ * This structure is not thread-safe.
+ */
+public class MetricStore {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricStore.class);
+
+ JobManagerMetricStore jobManager = new JobManagerMetricStore();
+ Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
+ Map<String, JobMetricStore> jobs = new HashMap<>();
+
+ /**
+ * Adds a metric to this MetricStore.
+ *
+ * @param name the metric identifier
+ * @param value the metric value
+ */
+ public void add(String name, Object value) {
+ TaskManagerMetricStore tm;
+ JobMetricStore job;
+ TaskMetricStore task;
+
+ try {
+ String[] components = name.split(":");
--- End diff --
Do we filter `:` out in user defined metric names?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2363
The test case `TaskManagerComponentsStartupShutdownTest.testComponentsStartupShutdown` fails on Travis.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75448166
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/AbstractMetricGroup.java ---
@@ -54,13 +53,19 @@
* return Counters, Gauges, etc to the code, to prevent exceptions in the monitored code.
* These metrics simply do not get reported any more, when created on a closed group.
*/
-public abstract class AbstractMetricGroup implements MetricGroup {
+public abstract class AbstractMetricGroup<A extends AbstractMetricGroup> implements MetricGroup {
--- End diff --
Sounds good. Haven't exclude the #2300 commit from the review. My bad.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75474843
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Nested data-structure to store metrics.
+ *
+ * This structure is not thread-safe.
+ */
+public class MetricStore {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricStore.class);
+
+ JobManagerMetricStore jobManager = new JobManagerMetricStore();
+ Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
+ Map<String, JobMetricStore> jobs = new HashMap<>();
+
+ /**
+ * Adds a metric to this MetricStore.
+ *
+ * @param name the metric identifier
+ * @param value the metric value
+ */
+ public void add(String name, Object value) {
--- End diff --
I think the object array representation (with the contained objects) is not as concise as the byte buffer representation (using your own serialization) because the standard Java serialization will write for every object the class name to identify it.
But again this falls imo into the category of premature optimization.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75346742
--- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/messages/Messages.scala ---
@@ -47,4 +47,21 @@ object Messages {
* @return The Acknowledge case object instance.
*/
def getAcknowledge(): Acknowledge.type = Acknowledge
+
+ /**
+ * Signals that the receiver (JobManager/TaskManager) should transmit a dump of the
+ * registered metrics.
+ *
+ * This message may be send regularly by the WebInterface.
+ */
+ case object MetricRequest
+
+ /**
+ * Accessor for the case object instance, to simplify Java interoperability.
+ *
+ * @return The MetricRequest case object instance.
+ */
+ def getRequestMetrics(): AnyRef = {
--- End diff --
Can we name this method like the message?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2363
But we still send metric data as strings encoded over the wire and have no checks that the histogram field order is actually correct, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75448294
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/ComponentMetricGroup.java ---
@@ -33,19 +36,42 @@
* group could for example include the task attempt number (more fine grained identification), or
* exclude it (for continuity of the namespace across failure and recovery).
*/
-public abstract class ComponentMetricGroup extends AbstractMetricGroup {
+public abstract class ComponentMetricGroup<P extends AbstractMetricGroup> extends AbstractMetricGroup<P> {
--- End diff --
The `JobMetricGroup` is annotated as `internal`. I was just wondering whether we should do the same for this metric group and the others (to which it applies) as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75344971
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/TaskManagerJobMetricGroup.java ---
@@ -61,9 +58,7 @@ public TaskManagerJobMetricGroup(
JobID jobId,
@Nullable String jobName) {
- super(registry, jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
-
- this.parent = checkNotNull(parent);
+ super(registry, checkNotNull(parent), jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
--- End diff --
shouldn't the `checkNotNull` be performed by the super constructor?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75447958
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricFetcher.java ---
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import akka.dispatch.OnSuccess;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.runtime.instance.ActorGateway;
+import org.apache.flink.runtime.instance.Instance;
+import org.apache.flink.runtime.messages.JobManagerMessages;
+import org.apache.flink.runtime.messages.Messages;
+import org.apache.flink.runtime.messages.webmonitor.JobDetails;
+import org.apache.flink.runtime.messages.webmonitor.MultipleJobsDetails;
+import org.apache.flink.runtime.messages.webmonitor.RequestJobDetails;
+import org.apache.flink.runtime.webmonitor.JobManagerRetriever;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.concurrent.ExecutionContext;
+import scala.concurrent.duration.Duration;
+import scala.concurrent.duration.FiniteDuration;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * The MetricFetcher can be used to fetch metrics from the JobManager and all registered TaskManagers.
+ *
+ * Metrics will only be fetched when {@link MetricFetcher#update()} is called, provided that a sufficient time since
+ * the last call has passed.
+ */
+public class MetricFetcher {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricFetcher.class);
+
+ private final JobManagerRetriever retriever;
+ private final ExecutionContext ctx;
+ private final FiniteDuration timeout = new FiniteDuration(Duration.create(ConfigConstants.DEFAULT_AKKA_ASK_TIMEOUT).toMillis(), TimeUnit.MILLISECONDS);
+
+ private MetricStore metrics = new MetricStore();
+
+ private long lastUpdateTime;
+
+ public MetricFetcher(JobManagerRetriever retriever, ExecutionContext ctx) {
+ this.retriever = Preconditions.checkNotNull(retriever);
+ this.ctx = Preconditions.checkNotNull(ctx);
+ }
+
+ /**
+ * Returns the MetricStore containing all stored metrics.
+ *
+ * @return MetricStore containing all stored metrics;
+ */
+ public MetricStore getMetricStore() {
+ return metrics;
+ }
+
+ /**
+ * This method can be used to signal this MetricFetcher that the metrics are still in use and should be updated.
+ */
+ public void update() {
+ synchronized (this) {
+ long currentTime = System.currentTimeMillis();
+ if (currentTime - lastUpdateTime > 10000) { // 10 seconds have passed since the last update
+ lastUpdateTime = currentTime;
+ fetchMetrics();
+ }
+ }
+ }
+
+ private void fetchMetrics() {
+ try {
+ Option<scala.Tuple2<ActorGateway, Integer>> jobManagerGatewayAndWebPort = retriever.getJobManagerGatewayAndWebPort();
+ if (jobManagerGatewayAndWebPort.isDefined()) {
+ ActorGateway jobManager = jobManagerGatewayAndWebPort.get()._1();
+
+ /**
+ * Remove all metrics that belong to a job that is not running and no longer archived.
+ */
+ jobManager.ask(new RequestJobDetails(true, true), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ MultipleJobsDetails details = (MultipleJobsDetails) result;
+ ArrayList<String> toRetain = new ArrayList<>();
+ for (JobDetails job : details.getRunningJobs()) {
+ toRetain.add(job.getJobId().toString());
+ }
+ for (JobDetails job : details.getFinishedJobs()) {
+ toRetain.add(job.getJobId().toString());
+ }
+ synchronized (metrics) {
+ metrics.jobs.keySet().retainAll(toRetain);
+ }
+ }
+ }, ctx);
+
+ /**
+ * Requests the metric dump from the job manager.
+ */
+ jobManager.ask(Messages.getRequestMetrics(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ addMetrics(result);
+ }
+ }, ctx);
+
+ /**
+ * We first request the list of all registered task managers from the job manager, and then
+ * request the respective metric dump from each task manager.
+ *
+ * All stored metrics that do not belong to a registered task manager will be removed.
+ */
+ jobManager.ask(JobManagerMessages.getRequestRegisteredTaskManagers(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ Iterable<Instance> taskManagers = ((JobManagerMessages.RegisteredTaskManagers) result).asJavaIterable();
+ List<String> activeTaskManagers = new ArrayList<>();
+ for (Instance taskManager : taskManagers) {
+ activeTaskManagers.add(taskManager.getId().toString());
+ taskManager.getActorGateway().ask(Messages.getRequestMetrics(), timeout)
+ .onSuccess(new OnSuccess<Object>() {
+ @Override
+ public void onSuccess(Object result) throws Throwable {
+ addMetrics(result);
+ }
+ }, ctx);
+ }
+ synchronized (metrics) { // remove all metrics belonging to unregistered task managers
+ metrics.taskManagers.keySet().retainAll(activeTaskManagers);
+ }
+ }
+ }, ctx);
+ }
+ } catch (Exception e) {
+ LOG.warn("Exception while fetching metrics.", e);
+ }
+ }
+
+ private void addMetrics(Object result) {
+ Object[] data = (Object[]) result;
+ for (int x = 0; x < data.length; ) {
+ synchronized (metrics) {
+ switch ((byte) data[x++]) {
+ case 0:
+ case 1:
--- End diff --
Yeah either that or sending the data in a more structured fashion as discussed below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r77527637
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -35,109 +46,111 @@
final Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
final Map<String, JobMetricStore> jobs = new HashMap<>();
- /**
- * Adds a metric to this MetricStore.
- *
- * @param name the metric identifier
- * @param value the metric value
- */
- public void add(String name, Object value) {
- TaskManagerMetricStore tm;
- JobMetricStore job;
- TaskMetricStore task;
-
+ public void add(MetricDump metric) {
try {
- String[] components = name.split(":");
- switch (components[0]) {
- /**
- * JobManagerMetricStore metric
- * format: 0:<user_scope>.<name>
- */
- case "0":
- jobManager.metrics.put(components[1], value);
- break;
- /**
- * TaskManager metric
- * format: 1:<tm_id>:<user_scope>.<name>
- */
- case "1":
- if (components.length != 3) {
- break;
- }
- tm = taskManagers.get(components[1]);
+ QueryScopeInfo info = metric.scopeInfo;
+ TaskManagerMetricStore tm;
+ JobMetricStore job;
+ TaskMetricStore task;
+
+ String name = info.scope.isEmpty()
+ ? metric.name
+ : info.scope + "." + metric.name;
+
+ if (name.isEmpty()) { // malformed transmission
+ return;
+ }
+
+ switch (info.getCategory()) {
+ case INFO_CATEGORY_JM:
+ addMetric(jobManager.metrics, name, metric);
+ case INFO_CATEGORY_TM:
+ String tmID = ((QueryScopeInfo.TaskManagerQueryScopeInfo) info).taskManagerID;
+ tm = taskManagers.get(tmID);
if (tm == null) {
tm = new TaskManagerMetricStore();
- taskManagers.put(components[1], tm);
+ taskManagers.put(tmID, tm);
}
- tm.metrics.put(components[2], value);
+ addMetric(tm.metrics, name, metric);
break;
- /**
- * Job metric
- * format: 2:<job_id>:<user_scope>.<name>
- */
- case "2":
- if (components.length != 3) {
- break;
- }
- job = jobs.get(components[1]);
+ case INFO_CATEGORY_JOB:
+ QueryScopeInfo.JobQueryScopeInfo jobInfo = (QueryScopeInfo.JobQueryScopeInfo) info;
+ job = jobs.get(jobInfo.jobID);
if (job == null) {
job = new JobMetricStore();
- jobs.put(components[1], job);
+ jobs.put(jobInfo.jobID, job);
}
- job.metrics.put(components[2], value);
+ addMetric(job.metrics, name, metric);
break;
- /**
- * Task metric
- * format: 3:<job_id>:<task_id>:<subtask_index>:<user_scope>.<name>
- *
- * As the WebInterface task metric queries currently do not account for subtasks we don't
- * divide by subtask and instead use the concatenation of subtask index and metric name as the name.
- */
- case "3":
- if (components.length != 5) {
- break;
- }
- job = jobs.get(components[1]);
+ case INFO_CATEGORY_TASK:
+ QueryScopeInfo.TaskQueryScopeInfo taskInfo = (QueryScopeInfo.TaskQueryScopeInfo) info;
+ job = jobs.get(taskInfo.jobID);
if (job == null) {
job = new JobMetricStore();
- jobs.put(components[1], job);
+ jobs.put(taskInfo.jobID, job);
}
- task = job.tasks.get(components[2]);
+ task = job.tasks.get(taskInfo.vertexID);
if (task == null) {
task = new TaskMetricStore();
- job.tasks.put(components[2], task);
+ job.tasks.put(taskInfo.vertexID, task);
}
- task.metrics.put(components[3] + "." + components[4], value);
+ /**
+ * As the WebInterface task metric queries currently do not account for subtasks we don't
+ * divide by subtask and instead use the concatenation of subtask index and metric name as the name.
+ */
+ addMetric(task.metrics, taskInfo.subtaskIndex + "." + name, metric);
break;
- /**
- * Operator metric
- * format: 4:<job_id>:<task_id>:<subtask_index>:<operator_name>:<user_scope>.<name>
- *
- * As the WebInterface does not account for operators (because it can't) we don't
- * divide by operator and instead use the concatenation of subtask index, operator name and metric name
- * as the name.
- */
- case "4":
- if (components.length != 6) {
- break;
- }
- job = jobs.get(components[1]);
+ case INFO_CATEGORY_OPERATOR:
+ QueryScopeInfo.OperatorQueryScopeInfo operatorInfo = (QueryScopeInfo.OperatorQueryScopeInfo) info;
+ job = jobs.get(operatorInfo.jobID);
if (job == null) {
job = new JobMetricStore();
- jobs.put(components[1], job);
+ jobs.put(operatorInfo.jobID, job);
}
- task = job.tasks.get(components[2]);
+ task = job.tasks.get(operatorInfo.vertexID);
if (task == null) {
task = new TaskMetricStore();
- job.tasks.put(components[2], task);
+ job.tasks.put(operatorInfo.vertexID, task);
}
- task.metrics.put(components[3] + "." + components[4] + "." + components[5], value);
+ /**
+ * As the WebInterface does not account for operators (because it can't) we don't
+ * divide by operator and instead use the concatenation of subtask index, operator name and metric name
+ * as the name.
+ */
+ addMetric(task.metrics, operatorInfo.subtaskIndex + "." + operatorInfo.operatorName + "." + name, metric);
break;
default:
- LOG.debug("Invalid metric name format: " + name);
+ LOG.debug("Invalid metric dump category: " + info.getCategory());
}
} catch (Exception e) {
- LOG.debug("Malformed metric name format: " + name);
+ LOG.debug("Malformed metric dump.", e);
+ }
+ }
+
+ private void addMetric(Map<String, Object> target, String name, MetricDump metric) {
+ switch (metric.getCategory()) {
+ case METRIC_CATEGORY_COUNTER:
--- End diff --
The same question applies here as well. Why don't we simply check what type the metric instance is?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r77528158
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricDumpSerialization.java ---
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.metrics.dump;
+
+import org.apache.commons.io.output.ByteArrayOutputStream;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.runtime.metrics.dump.QueryScopeInfo.INFO_CATEGORY_JM;
+import static org.apache.flink.runtime.metrics.dump.QueryScopeInfo.INFO_CATEGORY_JOB;
+import static org.apache.flink.runtime.metrics.dump.QueryScopeInfo.INFO_CATEGORY_OPERATOR;
+import static org.apache.flink.runtime.metrics.dump.QueryScopeInfo.INFO_CATEGORY_TASK;
+import static org.apache.flink.runtime.metrics.dump.QueryScopeInfo.INFO_CATEGORY_TM;
+
+/**
+ * Utility class for the serialization of metrics.
+ */
+public class MetricDumpSerialization {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricDumpSerialization.class);
+
+ private MetricDumpSerialization() {
+ }
+
+ // ===== Serialization =============================================================================================
+ public static class MetricDumpSerializer {
+ private ByteArrayOutputStream baos = new ByteArrayOutputStream(4096);
+ private DataOutputStream dos = new DataOutputStream(baos);
+
+ /**
+ * Serializes the given metrics and returns the resulting byte array.
+ *
+ * @param counters counters to serialize
+ * @param gauges gauges to serialize
+ * @param histograms histograms to serialize
+ * @return byte array containing the serialized metrics
+ * @throws IOException
+ */
+ public byte[] serialize(Map<Counter, Tuple2<QueryScopeInfo, String>> counters, Map<Gauge<?>, Tuple2<QueryScopeInfo, String>> gauges, Map<Histogram, Tuple2<QueryScopeInfo, String>> histograms) throws IOException {
+ baos.reset();
+ dos.writeInt(counters.size());
+ dos.writeInt(gauges.size());
+ dos.writeInt(histograms.size());
+
+ for (Map.Entry<Counter, Tuple2<QueryScopeInfo, String>> entry : counters.entrySet()) {
+ serializeMetricInfo(dos, entry.getValue().f0);
+ serializeString(dos, entry.getValue().f1);
+ serializeCounter(dos, entry.getKey());
+ }
+
+ for (Map.Entry<Gauge<?>, Tuple2<QueryScopeInfo, String>> entry : gauges.entrySet()) {
+ serializeMetricInfo(dos, entry.getValue().f0);
+ serializeString(dos, entry.getValue().f1);
+ serializeGauge(dos, entry.getKey());
+ }
+
+ for (Map.Entry<Histogram, Tuple2<QueryScopeInfo, String>> entry : histograms.entrySet()) {
+ serializeMetricInfo(dos, entry.getValue().f0);
+ serializeString(dos, entry.getValue().f1);
+ serializeHistogram(dos, entry.getKey());
+ }
+ return baos.toByteArray();
+ }
+
+ public void close() {
+ try {
+ dos.close();
+ } catch (Exception e) {
+ LOG.debug("Failed to close OutputStream.", e);
+ }
+ try {
+ baos.close();
+ } catch (Exception e) {
+ LOG.debug("Failed to close OutputStream.", e);
+ }
+ }
+ }
+
+ private static void serializeMetricInfo(DataOutputStream dos, QueryScopeInfo info) throws IOException {
+ serializeString(dos, info.scope);
+ dos.writeByte(info.getCategory());
+ switch (info.getCategory()) {
+ case INFO_CATEGORY_JM:
+ break;
+ case INFO_CATEGORY_TM:
+ String tmID = ((QueryScopeInfo.TaskManagerQueryScopeInfo) info).taskManagerID;
+ serializeString(dos, tmID);
+ break;
+ case INFO_CATEGORY_JOB:
+ QueryScopeInfo.JobQueryScopeInfo jobInfo = (QueryScopeInfo.JobQueryScopeInfo) info;
+ serializeString(dos, jobInfo.jobID);
+ break;
+ case INFO_CATEGORY_TASK:
+ QueryScopeInfo.TaskQueryScopeInfo taskInfo = (QueryScopeInfo.TaskQueryScopeInfo) info;
+ serializeString(dos, taskInfo.jobID);
+ serializeString(dos, taskInfo.vertexID);
+ dos.writeInt(taskInfo.subtaskIndex);
+ break;
+ case INFO_CATEGORY_OPERATOR:
+ QueryScopeInfo.OperatorQueryScopeInfo operatorInfo = (QueryScopeInfo.OperatorQueryScopeInfo) info;
+ serializeString(dos, operatorInfo.jobID);
+ serializeString(dos, operatorInfo.vertexID);
+ dos.writeInt(operatorInfo.subtaskIndex);
+ serializeString(dos, operatorInfo.operatorName);
+ break;
+ }
+ }
+
+ private static void serializeString(DataOutputStream dos, String string) throws IOException {
+ byte[] bytes = string.getBytes();
+ dos.writeInt(bytes.length);
+ dos.write(bytes);
+ }
+
+ private static void serializeCounter(DataOutputStream dos, Counter counter) throws IOException {
+ dos.writeLong(counter.getCount());
+ }
+
+ private static void serializeGauge(DataOutputStream dos, Gauge<?> gauge) throws IOException {
+ serializeString(dos, gauge.getValue().toString());
+ }
+
+ private static void serializeHistogram(DataOutputStream dos, Histogram histogram) throws IOException {
+ HistogramStatistics stat = histogram.getStatistics();
+
+ dos.writeLong(stat.getMin());
+ dos.writeLong(stat.getMax());
+ dos.writeDouble(stat.getMean());
+ dos.writeDouble(stat.getQuantile(0.5));
+ dos.writeDouble(stat.getStdDev());
+ dos.writeDouble(stat.getQuantile(0.75));
+ dos.writeDouble(stat.getQuantile(0.90));
+ dos.writeDouble(stat.getQuantile(0.95));
+ dos.writeDouble(stat.getQuantile(0.98));
+ dos.writeDouble(stat.getQuantile(0.99));
+ dos.writeDouble(stat.getQuantile(0.999));
+ }
+
+ // ===== Deserialization ===========================================================================================
--- End diff --
We usually use the following format for structural comments:
```
//-------------------------------------
// Foobar
//-------------------------------------
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r77527822
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricDump.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.metrics.dump;
+
+/**
+ * A container for a dumped metric that contains the scope, name and value(s) of the metric.
+ */
+public abstract class MetricDump {
+ /** Categories to be returned by {@link MetricDump#getCategory()} to avoid instanceof checks. */
+ public static final byte METRIC_CATEGORY_COUNTER = 0;
+ public static final byte METRIC_CATEGORY_GAUGE = 1;
+ public static final byte METRIC_CATEGORY_HISTOGRAM = 2;
+
+ /** The scope information for the stored metric. */
+ public final QueryScopeInfo scopeInfo;
+ /** The name of the stored metric. */
+ public final String name;
+
+ private MetricDump(QueryScopeInfo scopeInfo, String name) {
+ this.scopeInfo = scopeInfo;
+ this.name = name;
+ }
+
+ /**
+ * Returns the category for this MetricDump.
+ *
+ * @return category
+ */
+ public abstract byte getCategory();
--- End diff --
I think we don't need the explicit category information, because it is already encoded in the sub-types of `MetricDump`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75344502
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/ComponentMetricGroup.java ---
@@ -33,19 +36,42 @@
* group could for example include the task attempt number (more fine grained identification), or
* exclude it (for continuity of the namespace across failure and recovery).
*/
-public abstract class ComponentMetricGroup extends AbstractMetricGroup {
+public abstract class ComponentMetricGroup<P extends AbstractMetricGroup> extends AbstractMetricGroup<P> {
--- End diff --
Type parameter in the description is missing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75328166
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/AbstractMetricsHandler.java ---
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.runtime.instance.ActorGateway;
+import org.apache.flink.runtime.webmonitor.handlers.JsonFactory;
+import org.apache.flink.runtime.webmonitor.handlers.RequestHandler;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Map;
+
+/**
+ * Abstract request handler that returns a list of all available metrics or the values for a set of metrics.
+ *
+ * If the query parameters do not contain a "get" parameter the list of all metrics is returned.
+ * {@code {"available": [ { "name" : "X", "id" : "X" } ] } }
--- End diff --
`"name" : "X"` won't be written, will it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r77527854
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricDump.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.metrics.dump;
+
+/**
+ * A container for a dumped metric that contains the scope, name and value(s) of the metric.
+ */
+public abstract class MetricDump {
+ /** Categories to be returned by {@link MetricDump#getCategory()} to avoid instanceof checks. */
+ public static final byte METRIC_CATEGORY_COUNTER = 0;
+ public static final byte METRIC_CATEGORY_GAUGE = 1;
+ public static final byte METRIC_CATEGORY_HISTOGRAM = 2;
+
+ /** The scope information for the stored metric. */
+ public final QueryScopeInfo scopeInfo;
+ /** The name of the stored metric. */
+ public final String name;
+
+ private MetricDump(QueryScopeInfo scopeInfo, String name) {
+ this.scopeInfo = scopeInfo;
+ this.name = name;
--- End diff --
`checkNotNull` missing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r77540987
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -35,109 +46,111 @@
final Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
final Map<String, JobMetricStore> jobs = new HashMap<>();
- /**
- * Adds a metric to this MetricStore.
- *
- * @param name the metric identifier
- * @param value the metric value
- */
- public void add(String name, Object value) {
- TaskManagerMetricStore tm;
- JobMetricStore job;
- TaskMetricStore task;
-
+ public void add(MetricDump metric) {
try {
- String[] components = name.split(":");
- switch (components[0]) {
- /**
- * JobManagerMetricStore metric
- * format: 0:<user_scope>.<name>
- */
- case "0":
- jobManager.metrics.put(components[1], value);
- break;
- /**
- * TaskManager metric
- * format: 1:<tm_id>:<user_scope>.<name>
- */
- case "1":
- if (components.length != 3) {
- break;
- }
- tm = taskManagers.get(components[1]);
+ QueryScopeInfo info = metric.scopeInfo;
+ TaskManagerMetricStore tm;
+ JobMetricStore job;
+ TaskMetricStore task;
+
+ String name = info.scope.isEmpty()
+ ? metric.name
+ : info.scope + "." + metric.name;
+
+ if (name.isEmpty()) { // malformed transmission
+ return;
+ }
+
+ switch (info.getCategory()) {
+ case INFO_CATEGORY_JM:
--- End diff --
On the other hand, it does not seem too overly complicated to be not maintainable. With that in mind, my other comments are mainly obsolete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/2363
Fixed the failing test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r77538038
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -35,109 +46,111 @@
final Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
final Map<String, JobMetricStore> jobs = new HashMap<>();
- /**
- * Adds a metric to this MetricStore.
- *
- * @param name the metric identifier
- * @param value the metric value
- */
- public void add(String name, Object value) {
- TaskManagerMetricStore tm;
- JobMetricStore job;
- TaskMetricStore task;
-
+ public void add(MetricDump metric) {
try {
- String[] components = name.split(":");
- switch (components[0]) {
- /**
- * JobManagerMetricStore metric
- * format: 0:<user_scope>.<name>
- */
- case "0":
- jobManager.metrics.put(components[1], value);
- break;
- /**
- * TaskManager metric
- * format: 1:<tm_id>:<user_scope>.<name>
- */
- case "1":
- if (components.length != 3) {
- break;
- }
- tm = taskManagers.get(components[1]);
+ QueryScopeInfo info = metric.scopeInfo;
+ TaskManagerMetricStore tm;
+ JobMetricStore job;
+ TaskMetricStore task;
+
+ String name = info.scope.isEmpty()
+ ? metric.name
+ : info.scope + "." + metric.name;
+
+ if (name.isEmpty()) { // malformed transmission
+ return;
+ }
+
+ switch (info.getCategory()) {
+ case INFO_CATEGORY_JM:
--- End diff --
eh, seemed like the proper way of handling it. Also, (up to) 4 comparisons vs a jump.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/2363
Regarding hierarchy: I'm close to being done with a container for the scope information.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75347741
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricQueryService.java ---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.metrics;
+
+import akka.actor.ActorRef;
+import akka.actor.ActorSystem;
+import akka.actor.Props;
+import akka.actor.UntypedActor;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The MetricQueryService creates a key-value representation of all metrics currently registered with Flink when queried.
+ *
+ * It is realized as an actor and receives the following messages:
+ * - {@code Tuple3<String, Metric, AbstractMetricGroup> => Notification of added Metric}
+ * - {@code Metric => Notification of removed Metric}
+ * - {@code ActorRef => Query for metric dump}
+ */
+public class MetricQueryService extends UntypedActor {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricQueryService.class);
+
+ public static final byte CATEGORY_COUNTER = 0;
+ public static final byte CATEGORY_GAUGE = 1;
+ public static final byte CATEGORY_HISTOGRAM = 2;
+
+ private final Map<Gauge<?>, String> gauges = new HashMap<>();
+ private final Map<Counter, String> counters = new HashMap<>();
+ private final Map<Histogram, String> histograms = new HashMap<>();
+
+ private final List<Metric> toRemove = new ArrayList<>();
+
+ @Override
+ public void onReceive(Object message) throws Exception {
+ try {
+ if (message instanceof Tuple3) { // add metric
+ Tuple3<String, Metric, AbstractMetricGroup> tuple = (Tuple3<String, Metric, AbstractMetricGroup>) message;
+
+ String metricName = tuple.f0;
+ Metric metric = tuple.f1;
+ AbstractMetricGroup group = tuple.f2;
+
+ String name = group.getQueryServiceMetricIdentifier(metricName, new CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return input.replaceAll("[ :.,]", "_");
+ }
+ });
+
+ if (metric instanceof Counter) {
+ counters.put((Counter) metric, name);
+ } else if (metric instanceof Gauge) {
+ gauges.put((Gauge<?>) metric, name);
+ } else if (metric instanceof Histogram) {
+ histograms.put((Histogram) metric, name);
+ }
+ } else if (message instanceof Metric) { // remove metric
+ toRemove.add((Metric) message);
+ } else if (message instanceof ActorRef) { // create dump
+ Object[] dump = createDump();
+ ((ActorRef) message).tell(dump, getSelf());
+ removeMetrics();
+ } else {
+ LOG.warn("MetricQueryServiceActor received an invalid message. " + message.toString());
+ }
+ } catch (Exception e) {
+ LOG.warn("An exception occurred while processing a message.", e);
+ }
+ }
+
+ private Object[] createDump() {
+ Object[] metrics = new Object[(this.counters.size() + this.gauges.size()) * 3 + this.histograms.size() * 13];
+
+ int index = 0;
+
+ // dump counters
+ for (Map.Entry<Counter, String> counter : this.counters.entrySet()) {
+ metrics[index++] = CATEGORY_COUNTER;
+ metrics[index++] = counter.getValue();
+ metrics[index++] = counter.getKey().getCount();
+ }
+ // dump gauges
+ for (Map.Entry<Gauge<?>, String> gauge : this.gauges.entrySet()) {
+ metrics[index++] = CATEGORY_GAUGE;
+ metrics[index++] = gauge.getValue();
+ metrics[index++] = gauge.getKey().getValue().toString();
+ }
+ // dump histograms
+ for (Map.Entry<Histogram, String> histogram : this.histograms.entrySet()) {
+ HistogramStatistics stat = histogram.getKey().getStatistics();
+
+ metrics[index++] = CATEGORY_HISTOGRAM;
+ metrics[index++] = histogram.getValue();
+ metrics[index++] = stat.getMin();
+ metrics[index++] = stat.getMax();
+ metrics[index++] = stat.getMean();
+ metrics[index++] = stat.getQuantile(0.5);
+ metrics[index++] = stat.getStdDev();
+ metrics[index++] = stat.getQuantile(0.75);
+ metrics[index++] = stat.getQuantile(0.9);
+ metrics[index++] = stat.getQuantile(0.95);
+ metrics[index++] = stat.getQuantile(0.98);
+ metrics[index++] = stat.getQuantile(0.99);
+ metrics[index++] = stat.getQuantile(0.999);
+ }
+ return metrics;
+ }
+
+ private void removeMetrics() {
+ for (Metric metric : toRemove) {
+ if (metric instanceof Counter) {
+ this.counters.remove(metric);
+ } else if (metric instanceof Gauge) {
+ this.gauges.remove(metric);
+ } else if (metric instanceof Histogram) {
+ this.histograms.remove(metric);
+ }
+ }
+ }
+
+ /**
+ * Starts the MetricQueryService actor in the given actor system.
+ *
+ * @param actorSystem The actor system running the MetricQueryService
+ * @return actor reference to the MetricQueryService
+ */
+ public static ActorRef startMetricQueryService(ActorSystem actorSystem) {
+ return actorSystem.actorOf(Props.create(MetricQueryService.class));
--- End diff --
Maybe we could give the metric query service a concrete name. Then we would not have to ask the `JobManager` about the metrics. Based on the `JobManager` and `TaskManager` addresses, we could deduce the query metric service's address and ask it directly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75644101
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/TaskManagerJobMetricGroup.java ---
@@ -61,9 +58,7 @@ public TaskManagerJobMetricGroup(
JobID jobId,
@Nullable String jobName) {
- super(registry, jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
-
- this.parent = checkNotNull(parent);
+ super(registry, checkNotNull(parent), jobId, jobName, scopeFormat.formatScope(parent, jobId, jobName));
--- End diff --
fair enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r77527444
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -35,109 +46,111 @@
final Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
final Map<String, JobMetricStore> jobs = new HashMap<>();
- /**
- * Adds a metric to this MetricStore.
- *
- * @param name the metric identifier
- * @param value the metric value
- */
- public void add(String name, Object value) {
- TaskManagerMetricStore tm;
- JobMetricStore job;
- TaskMetricStore task;
-
+ public void add(MetricDump metric) {
try {
- String[] components = name.split(":");
- switch (components[0]) {
- /**
- * JobManagerMetricStore metric
- * format: 0:<user_scope>.<name>
- */
- case "0":
- jobManager.metrics.put(components[1], value);
- break;
- /**
- * TaskManager metric
- * format: 1:<tm_id>:<user_scope>.<name>
- */
- case "1":
- if (components.length != 3) {
- break;
- }
- tm = taskManagers.get(components[1]);
+ QueryScopeInfo info = metric.scopeInfo;
+ TaskManagerMetricStore tm;
+ JobMetricStore job;
+ TaskMetricStore task;
+
+ String name = info.scope.isEmpty()
+ ? metric.name
+ : info.scope + "." + metric.name;
+
+ if (name.isEmpty()) { // malformed transmission
+ return;
+ }
+
+ switch (info.getCategory()) {
+ case INFO_CATEGORY_JM:
--- End diff --
What's the benefit of having an explicit type field over using `instanceof`? I think encoding the type via the actual type has the advantage that you don't mix up classes with wrong category types.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/2363
@tillrohrmann I've addressed most of your comments. Excluded are calling `checkNotNull` inside `formatScope` and the serialization format.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2363: [FLINK-4389] Expose metrics to WebFrontend
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2363#discussion_r75450661
--- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/metrics/MetricStore.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.webmonitor.metrics;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Nested data-structure to store metrics.
+ *
+ * This structure is not thread-safe.
+ */
+public class MetricStore {
+ private static final Logger LOG = LoggerFactory.getLogger(MetricStore.class);
+
+ JobManagerMetricStore jobManager = new JobManagerMetricStore();
+ Map<String, TaskManagerMetricStore> taskManagers = new HashMap<>();
+ Map<String, JobMetricStore> jobs = new HashMap<>();
+
+ /**
+ * Adds a metric to this MetricStore.
+ *
+ * @param name the metric identifier
+ * @param value the metric value
+ */
+ public void add(String name, Object value) {
--- End diff --
You're right, I've combined two things here. But B) also depends a little bit on A) (at least implementation wise)
Concerning A) I'm not quite sure whether this is not a classical case of premature optimization. Have we actually measured the impact of the other approach? It would be good to know what it costs us. I would assume that the overhead is negligible.
Furthermore, I'm think you can also achieve an efficient serialization without too many short-living objects if you don't send strings around. First of all, we could serialize all metrics into a continuous byte buffer. Then we wouldn't have the object array overhead. Next we could use our own serializers to store the different object types efficiently (e.g. byte encoding for different metric types). On the server side you would only create the metric name objects once and that's all. Only upon deserialization you would have to create a new set of these objects. But that is ok since it happens on the web server side. The important thing is, though, to have a serialization component which is responsible for serialization and deserialization. That will avoid spreading the encoding over multiple places.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---