You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "hardikbajaj (via GitHub)" <gi...@apache.org> on 2023/06/01 14:08:12 UTC
[GitHub] [druid] hardikbajaj opened a new pull request, #14359: Added new SysMonitorOshi v0 using Oshi library
hardikbajaj opened a new pull request, #14359:
URL: https://github.com/apache/druid/pull/14359
<!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
<!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
Fixes #14157. (SysMonitor on ARM in roadmap)
<!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
<!-- If you are a committer, follow the PR action item checklist for committers:
https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
### Description
SysMonitor uses [Sigar](https://github.com/hyperic/sigar) library, which not only is not maintained and doesn't run on ARM64 arch. A JNA based [oshi](https://github.com/oshi/oshi) library was proposed by @abhagraw in slack community which is well maintained, used by projects like Apache flink, and also works well on ARM architecture too. Here I introduce v0 of `SysMonitorOshi`, oshi based SysMonitor for Druid.
This monitor monitors most of the metrics emitted by `SysMonitor.java` but is a v0 implementation which just consumes metrics emitted by Oshi directly, without modifying/mapping it.
<!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
<!-- Describe your patch: what did you change in code? How did you fix the problem? -->
<!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
#### Fixed the bug ...
SysMonitor not usable in ARM architecture
<!--
In each section, please describe design decisions made, including:
- Choice of algorithms
- Behavioral aspects. What configuration values are acceptable? How are corner cases and error conditions handled, such as when there are insufficient resources?
- Class organization and design (how the logic is split between classes, inheritance, composition, design patterns)
- Method organization and design (how the logic is split between methods, parameters and return types)
- Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
-->
<!-- It's good to describe an alternative design (or mention an alternative name) for every design (or naming) decision point and compare the alternatives with the designs that you've implemented (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
<!-- If there was a discussion of the design of the feature implemented in this PR elsewhere (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), link to that discussion from this PR description and explain what have changed in your final design compared to your original proposal or the consensus version in the end of the discussion. If something hasn't changed since the original discussion, you can omit a detailed discussion of those aspects of the design here, perhaps apart from brief mentioning for the sake of readability of this PR description. -->
<!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
#### Release note
<!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users.
If your change doesn't have end user impact, you can skip this section.
For tips about how to write a good release note, see [Release notes](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#release-notes).
-->
<hr>
##### Key changed/added classes in this PR
* `SysMonitorOshi.java`
* `NoopSysMonitorOshi.java`
* `SysMonitorOshiTest.java`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [x] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [x] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1213971750
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(SysMonitorOshi.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public SysMonitorOshi()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/mem/max", mem.getTotal(),
Review Comment:
this will double the volume of metrics and there are tools out there already tracking these metrics. If we change the name, then those tools need to adjust. That will be painful. so let's retain the same names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1220852618
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", type)
+ .setDimension("fsSysTypeName", fs.getType())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ // Difference b/w metrics of two consecutive values. It tells Δmetric (increase/decrease in metrics value)
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+
+ final Map<String, Long> stats = diff.to(
+ disk.getName(),
+ ImmutableMap.<String, Long>builder()
+ .put("sys/disk/read/size", disk.getReads())
+ .put("sys/disk/read/count", disk.getReadBytes())
+ .put("sys/disk/write/size", disk.getWrites())
+ .put("sys/disk/write/count", disk.getWriteBytes())
+ .put("sys/disk/queue", disk.getCurrentQueueLength())
+ .put("sys/disk/transferTime", disk.getTransferTime())
Review Comment:
Acknowledged, thanks for pointing this out. I missed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1231806482
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", type)
+ .setDimension("fsSysTypeName", fs.getType())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ // Difference b/w metrics of two consecutive values. It tells Δmetric (increase/decrease in metrics value)
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+
+ final Map<String, Long> stats = diff.to(
+ disk.getName(),
+ ImmutableMap.<String, Long>builder()
+ .put("sys/disk/read/size", disk.getReads())
+ .put("sys/disk/read/count", disk.getReadBytes())
+ .put("sys/disk/write/size", disk.getWrites())
+ .put("sys/disk/write/count", disk.getWriteBytes())
+ .put("sys/disk/queue", disk.getCurrentQueueLength())
+ .put("sys/disk/transferTime", disk.getTransferTime())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("diskName", disk.getName())
+ .setDimension("diskModel", disk.getModel());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+
+ private class NetStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<NetworkIF> networkIFS = hal.getNetworkIFs();
+ for (NetworkIF net : networkIFS) {
+ final String name = net.getName();
+ for (String addr : net.getIPv4addr()) {
+ if (netAddressBlacklist.contains(addr)) {
+ net = null;
+ log.debug("Not monitoring network stats for ip addr: [%s], network name: [%s]", addr, name);
+ break;
+ }
+ }
+ if (net != null) {
+
+ final Map<String, Long> stats = diff.to(
+ name,
+ ImmutableMap.<String, Long>builder()
+ .put("sys/net/read/size", net.getBytesRecv())
+ .put("sys/net/read/packets", net.getPacketsRecv())
+ .put("sys/net/read/errors", net.getInErrors())
+ .put("sys/net/read/dropped", net.getInDrops())
+ .put("sys/net/write/size", net.getBytesSent())
+ .put("sys/net/write/packets", net.getPacketsSent())
+ .put("sys/net/write/errors", net.getOutErrors())
+ .put("sys/net/write/collisions", net.getCollisions())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("netName", net.getName())
+ .setDimension("netAddress", Arrays.toString(net.getIPv4addr()))
Review Comment:
Discussed and implemented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on PR #14359:
URL: https://github.com/apache/druid/pull/14359#issuecomment-1599018749
Merged. Appreciate the persistency @hardikbajaj :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1213947465
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(SysMonitorOshi.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public SysMonitorOshi()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/mem/max", mem.getTotal(),
Review Comment:
Yeah, that was my concern. If someone uses SysMonitor till Oshi monitor is fully documented and tested.
I thought we'll change back to Sys once SysMonitor is deprecated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1214074465
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(SysMonitorOshi.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public SysMonitorOshi()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/mem/max", mem.getTotal(),
+ "sysO/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sysO/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/swap/pageIn", currPageIn - prevPageIn,
+ "sysO/swap/pageOut", currPageOut - prevPageOut,
+ "sysO/swap/max", swap.getSwapTotal(),
+ "sysO/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ if (fsTypeWhitelist.contains(fs.getType())) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sysO/fs/max", fs.getTotalSpace())
+ .put("sysO/fs/used", fs.getUsableSpace())
+ .put("sysO/fs/files/count", fs.getTotalInodes())
+ .put("sysO/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getName())
+ .setDimension("fsDirName", fs.getVolume())
+ .setDimension("fsTypeName", fs.getType())
+ .setDimension("fsSysTypeName", fs.getDescription())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sysO/disk/read/size", disk.getReads())
+ .put("sysO/disk/read/count", disk.getReadBytes())
+ .put("sysO/disk/write/size", disk.getWrites())
+ .put("sysO/disk/write/count", disk.getWriteBytes())
+ .put("sysO/disk/queue", disk.getCurrentQueueLength())
+ .put("sysO/disk/transferTime", disk.getTransferTime())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("diskName", disk.getName())
+ .setDimension("diskModel", disk.getModel());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class NetStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<NetworkIF> networkIFS = hal.getNetworkIFs();
+ for (NetworkIF net : networkIFS) {
+ final String name = net.getName();
+ for (String addr : net.getIPv4addr()) {
+ if (netAddressBlacklist.contains(addr)) {
+ net = null;
+ log.debug("Not monitoring network stats for ip addr: [%s], network name: [%s]", addr, name);
+ break;
+ }
+ }
+ if (net != null) {
+
+ final Map<String, Long> stats = diff.to(
+ name,
+ ImmutableMap.<String, Long>builder()
+ .put("sysO/net/read/size", net.getBytesRecv())
+ .put("sysO/net/read/packets", net.getPacketsRecv())
+ .put("sysO/net/read/errors", net.getInErrors())
+ .put("sysO/net/read/dropped", net.getInDrops())
+ .put("sysO/net/write/size", net.getBytesSent())
+ .put("sysO/net/write/packets", net.getPacketsSent())
+ .put("sysO/net/write/errors", net.getOutErrors())
+ .put("sysO/net/write/collisions", net.getCollisions())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("netName", net.getName())
+ .setDimension("netAddress", net.getMacaddr())
+ .setDimension("netHwaddr", Arrays.toString(net.getIPv6addr()));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+ }
+
+ private class CpuStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ CentralProcessor processor = hal.getProcessor();
+ long[][] procTicks = processor.getProcessorCpuLoadTicks();
+ for (int i = 0; i < procTicks.length; ++i) {
+ final String name = Integer.toString(i);
+ long[] ticks = procTicks[i];
+ long user = ticks[TickType.USER.getIndex()];
+ long nice = ticks[TickType.NICE.getIndex()];
+ long sys = ticks[TickType.SYSTEM.getIndex()];
+ long idle = ticks[TickType.IDLE.getIndex()];
+ long iowait = ticks[TickType.IOWAIT.getIndex()];
+ long irq = ticks[TickType.IRQ.getIndex()];
+ long softirq = ticks[TickType.SOFTIRQ.getIndex()];
+ long steal = ticks[TickType.STEAL.getIndex()];
+ long totalCpu = user + nice + sys + idle + iowait + irq + softirq + steal;
+ final Map<String, Long> stats = diff.to(
+ name,
+ ImmutableMap.<String, Long>builder()
+ .put("user", user) // user = Δuser / Δtotal
+ .put("sys", sys) // sys = Δsys / Δtotal
+ .put("nice", nice) // nice = Δnice / Δtotal
+ .put("wait", iowait) // wait = Δwait / Δtotal
+ .put("irq", irq) // irq = Δirq / Δtotal
+ .put("softIrq", softirq) // softIrq = ΔsoftIrq / Δtotal
+ .put("stolen", steal) // stolen = Δstolen / Δtotal
+ .put("_total", totalCpu) // (not reported)
+ .build()
+ );
+ if (stats != null) {
+ final long total = stats.remove("_total");
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("cpuName", name)
+ .setDimension("cpuTime", entry.getKey());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ if (total != 0) {
+ // prevent divide by 0 exception and don't emit such events
+ emitter.emit(builder.build("sysO/cpu", entry.getValue() * 100 / total)); // [0,100]
+ }
+
+ }
+ }
+ }
+ }
+ }
+
+ private class SysStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+
+ long uptime = os.getSystemUptime();
+
+ final Map<String, Number> stats = ImmutableMap.of(
+ "sysO/uptime", uptime
+ );
+ for (Map.Entry<String, Number> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ CentralProcessor processor = hal.getProcessor();
+ double[] la = processor.getSystemLoadAverage(3);
+
+ if (la != null) {
+ final Map<String, Number> statsCpuLoadAverage = ImmutableMap.of(
+ "sysO/la/1", la[0],
+ "sysO/la/5", la[1],
+ "sysO/la/15", la[2]
+ );
+ for (Map.Entry<String, Number> entry : statsCpuLoadAverage.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class TcpStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+
+ InternetProtocolStats ipstats = os.getInternetProtocolStats();
+ InternetProtocolStats.TcpStats tcpv4 = ipstats.getTCPv4Stats();
+
+ if (tcpv4 != null) {
+ final Map<String, Long> stats = diff.to(
+ "tcpv4", ImmutableMap.<String, Long>builder()
+ .put("sysO/tcpv4/activeOpens", tcpv4.getConnectionsActive())
+ .put("sysO/tcpv4/passiveOpens", tcpv4.getConnectionsPassive())
+ .put("sysO/tcpv4/attemptFails", tcpv4.getConnectionFailures())
+ .put("sysO/tcpv4/estabResets", tcpv4.getConnectionsReset())
+ .put("sysO/tcpv4/in/segs", tcpv4.getSegmentsReceived())
+ .put("sysO/tcpv4/in/errs", tcpv4.getInErrors())
+ .put("sysO/tcpv4/out/segs", tcpv4.getSegmentsSent())
+ .put("sysO/tcpv4/out/rsts", tcpv4.getOutResets())
+ .put("sysO/tcpv4/retrans/segs", tcpv4.getSegmentsRetransmitted())
+ .build()
+ );
+ if (stats != null) {
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ InternetProtocolStats.TcpStats tcpv6 = ipstats.getTCPv6Stats();
+
+ if (tcpv6 != null) {
+ final Map<String, Long> stats = diff.to(
+ "tcpv6", ImmutableMap.<String, Long>builder()
+ .put("sysO/tcpv6/activeOpens", tcpv6.getConnectionsActive())
+ .put("sysO/tcpv6/passiveOpens", tcpv6.getConnectionsPassive())
+ .put("sysO/tcpv6/attemptFails", tcpv6.getConnectionFailures())
+ .put("sysO/tcpv6/estabResets", tcpv6.getConnectionsReset())
+ .put("sysO/tcpv6/in/segs", tcpv6.getSegmentsReceived())
+ .put("sysO/tcpv6/in/errs", tcpv6.getInErrors())
+ .put("sysO/tcpv6/out/segs", tcpv6.getSegmentsSent())
+ .put("sysO/tcpv6/out/rsts", tcpv6.getOutResets())
+ .put("sysO/tcpv6/retrans/segs", tcpv6.getSegmentsRetransmitted())
+ .build()
+ );
+ if (stats != null) {
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
Review Comment:
We can leave v6 metrics out for now. since that seems to be increasing the metric volume unnecessarily. We also have these metrics missing in Oshi
```
Only in Sigar | .sys/mem/actual/used
Only in Sigar | .sys/net/write/overruns
Only in Sigar | .sys/net/read/frame
Only in Sigar | .sys/net/read/overruns
Only in Sigar | .sys/net/outbound
Only in Sigar | .sys/mem/actual/free
Only in Sigar | .sys/net/write/dropped
Only in Sigar | .sys/net/inbound
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on PR #14359:
URL: https://github.com/apache/druid/pull/14359#issuecomment-1573224906
I also unlinked the community roadmap issue from this PR so that the issue doesn't get closed after this PR is merged :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1220853710
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", type)
+ .setDimension("fsSysTypeName", fs.getType())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ // Difference b/w metrics of two consecutive values. It tells Δmetric (increase/decrease in metrics value)
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+
+ final Map<String, Long> stats = diff.to(
+ disk.getName(),
+ ImmutableMap.<String, Long>builder()
+ .put("sys/disk/read/size", disk.getReads())
+ .put("sys/disk/read/count", disk.getReadBytes())
+ .put("sys/disk/write/size", disk.getWrites())
+ .put("sys/disk/write/count", disk.getWriteBytes())
+ .put("sys/disk/queue", disk.getCurrentQueueLength())
+ .put("sys/disk/transferTime", disk.getTransferTime())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("diskName", disk.getName())
+ .setDimension("diskModel", disk.getModel());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+
+ private class NetStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<NetworkIF> networkIFS = hal.getNetworkIFs();
+ for (NetworkIF net : networkIFS) {
+ final String name = net.getName();
+ for (String addr : net.getIPv4addr()) {
Review Comment:
yes, it's supposed to return a list of Ipv4 addresses. Returning an empty array in some cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1231857782
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", type)
+ .setDimension("fsSysTypeName", fs.getType())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
Review Comment:
Going ahead with Disk metrics being emitted and left a comment if partition level metrics needs to be emitted in future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1221058047
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
Review Comment:
fsTypeWhitelist right ? Yeah you're right, now with local SysTYpe filter, we would not be needing this anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1234703689
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -313,8 +313,9 @@ public void emit(ServiceEmitter emitter)
for (String addr : net.getIPv4addr()) {
if (!NET_ADDRESS_BLACKLIST.contains(addr)) {
// Only emit metrics for non black-listed ip addresses
+ String mapKey = name + "_" + addr; // Network_Name_IPV4 address as key, ex: wifi_192.1.0.1 to uniquely identify the dimension
Review Comment:
looks like the tests were indeed helpful :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on PR #14359:
URL: https://github.com/apache/druid/pull/14359#issuecomment-1598071338
Approved and will merge once CI passes. Thank you @hardikbajaj for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1217018323
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(SysMonitorOshi.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public SysMonitorOshi()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/mem/max", mem.getTotal(),
+ "sysO/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sysO/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/swap/pageIn", currPageIn - prevPageIn,
+ "sysO/swap/pageOut", currPageOut - prevPageOut,
+ "sysO/swap/max", swap.getSwapTotal(),
+ "sysO/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ if (fsTypeWhitelist.contains(fs.getType())) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sysO/fs/max", fs.getTotalSpace())
+ .put("sysO/fs/used", fs.getUsableSpace())
+ .put("sysO/fs/files/count", fs.getTotalInodes())
+ .put("sysO/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getName())
Review Comment:
@abhishekagarwal87 Marking it unresolved until we figure out best way to get fsTypeName.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1221055092
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
Review Comment:
doing total - usable. Thanks for pointing this out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 merged pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 merged PR #14359:
URL: https://github.com/apache/druid/pull/14359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1232082353
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+
+ private final SystemInfo si;
+ private final HardwareAbstractionLayer hal;
+ private final OperatingSystem os;
+ private static final List<String> NET_ADDRESS_BLACKLIST = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.si = new SystemInfo();
+ this.hal = si.getHardware();
+ this.os = si.getOperatingSystem();
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max",
+ mem.getTotal(),
+ "sys/mem/used",
+ mem.getTotal() - mem.getAvailable(),
+ // This is total actual memory used, not including cache and buffer memory
+ "sys/mem/free",
+ mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores(true)) { // get only local file store : true
+
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getTotalSpace() - fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", "local") // Only local disk types displayed
Review Comment:
lets remove this dimension.
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+
+ private final SystemInfo si;
+ private final HardwareAbstractionLayer hal;
+ private final OperatingSystem os;
+ private static final List<String> NET_ADDRESS_BLACKLIST = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.si = new SystemInfo();
+ this.hal = si.getHardware();
+ this.os = si.getOperatingSystem();
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ // Create an object with mocked systemInfo for testing purposes
+ public OshiSysMonitor(SystemInfo systemInfo)
+ {
+ super("metrics");
+ this.dimensions = ImmutableMap.of();
+
+ this.si = systemInfo;
+ this.hal = si.getHardware();
+ this.os = si.getOperatingSystem();
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
Review Comment:
You can create these outside and retain references in the main class. Then you can get rid of monitorStats method that takes an array index. And you can have methods specific for each stat type. E.g.
you will have monitorMemStats, monitorSwapStats, etc. in your class and then you can invoke those from test.
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+
+ private final SystemInfo si;
+ private final HardwareAbstractionLayer hal;
+ private final OperatingSystem os;
+ private static final List<String> NET_ADDRESS_BLACKLIST = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.si = new SystemInfo();
+ this.hal = si.getHardware();
+ this.os = si.getOperatingSystem();
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ // Create an object with mocked systemInfo for testing purposes
+ public OshiSysMonitor(SystemInfo systemInfo)
+ {
+ super("metrics");
+ this.dimensions = ImmutableMap.of();
+
+ this.si = systemInfo;
+ this.hal = si.getHardware();
+ this.os = si.getOperatingSystem();
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ // Emit stats for a particular stat(mem, swap, filestore, etc) from statsList for testing
+ public void monitorStats(int statIndex, ServiceEmitter emitter)
Review Comment:
Hmm. a test doesn't know what the index for mem stats is. It works but seems a bit brittle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1214080715
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(SysMonitorOshi.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public SysMonitorOshi()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/mem/max", mem.getTotal(),
+ "sysO/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sysO/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/swap/pageIn", currPageIn - prevPageIn,
+ "sysO/swap/pageOut", currPageOut - prevPageOut,
+ "sysO/swap/max", swap.getSwapTotal(),
+ "sysO/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ if (fsTypeWhitelist.contains(fs.getType())) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sysO/fs/max", fs.getTotalSpace())
+ .put("sysO/fs/used", fs.getUsableSpace())
+ .put("sysO/fs/files/count", fs.getTotalInodes())
+ .put("sysO/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getName())
+ .setDimension("fsDirName", fs.getVolume())
+ .setDimension("fsTypeName", fs.getType())
+ .setDimension("fsSysTypeName", fs.getDescription())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sysO/disk/read/size", disk.getReads())
+ .put("sysO/disk/read/count", disk.getReadBytes())
+ .put("sysO/disk/write/size", disk.getWrites())
+ .put("sysO/disk/write/count", disk.getWriteBytes())
+ .put("sysO/disk/queue", disk.getCurrentQueueLength())
+ .put("sysO/disk/transferTime", disk.getTransferTime())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("diskName", disk.getName())
+ .setDimension("diskModel", disk.getModel());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class NetStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<NetworkIF> networkIFS = hal.getNetworkIFs();
+ for (NetworkIF net : networkIFS) {
+ final String name = net.getName();
+ for (String addr : net.getIPv4addr()) {
+ if (netAddressBlacklist.contains(addr)) {
+ net = null;
+ log.debug("Not monitoring network stats for ip addr: [%s], network name: [%s]", addr, name);
+ break;
+ }
+ }
+ if (net != null) {
+
+ final Map<String, Long> stats = diff.to(
+ name,
+ ImmutableMap.<String, Long>builder()
+ .put("sysO/net/read/size", net.getBytesRecv())
+ .put("sysO/net/read/packets", net.getPacketsRecv())
+ .put("sysO/net/read/errors", net.getInErrors())
+ .put("sysO/net/read/dropped", net.getInDrops())
+ .put("sysO/net/write/size", net.getBytesSent())
+ .put("sysO/net/write/packets", net.getPacketsSent())
+ .put("sysO/net/write/errors", net.getOutErrors())
+ .put("sysO/net/write/collisions", net.getCollisions())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("netName", net.getName())
+ .setDimension("netAddress", net.getMacaddr())
+ .setDimension("netHwaddr", Arrays.toString(net.getIPv6addr()));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+ }
+
+ private class CpuStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ CentralProcessor processor = hal.getProcessor();
+ long[][] procTicks = processor.getProcessorCpuLoadTicks();
+ for (int i = 0; i < procTicks.length; ++i) {
+ final String name = Integer.toString(i);
+ long[] ticks = procTicks[i];
+ long user = ticks[TickType.USER.getIndex()];
+ long nice = ticks[TickType.NICE.getIndex()];
+ long sys = ticks[TickType.SYSTEM.getIndex()];
+ long idle = ticks[TickType.IDLE.getIndex()];
+ long iowait = ticks[TickType.IOWAIT.getIndex()];
+ long irq = ticks[TickType.IRQ.getIndex()];
+ long softirq = ticks[TickType.SOFTIRQ.getIndex()];
+ long steal = ticks[TickType.STEAL.getIndex()];
+ long totalCpu = user + nice + sys + idle + iowait + irq + softirq + steal;
+ final Map<String, Long> stats = diff.to(
+ name,
+ ImmutableMap.<String, Long>builder()
+ .put("user", user) // user = Δuser / Δtotal
+ .put("sys", sys) // sys = Δsys / Δtotal
+ .put("nice", nice) // nice = Δnice / Δtotal
+ .put("wait", iowait) // wait = Δwait / Δtotal
+ .put("irq", irq) // irq = Δirq / Δtotal
+ .put("softIrq", softirq) // softIrq = ΔsoftIrq / Δtotal
+ .put("stolen", steal) // stolen = Δstolen / Δtotal
+ .put("_total", totalCpu) // (not reported)
+ .build()
+ );
+ if (stats != null) {
+ final long total = stats.remove("_total");
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("cpuName", name)
+ .setDimension("cpuTime", entry.getKey());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ if (total != 0) {
+ // prevent divide by 0 exception and don't emit such events
+ emitter.emit(builder.build("sysO/cpu", entry.getValue() * 100 / total)); // [0,100]
+ }
+
+ }
+ }
+ }
+ }
+ }
+
+ private class SysStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+
+ long uptime = os.getSystemUptime();
+
+ final Map<String, Number> stats = ImmutableMap.of(
+ "sysO/uptime", uptime
+ );
+ for (Map.Entry<String, Number> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ CentralProcessor processor = hal.getProcessor();
+ double[] la = processor.getSystemLoadAverage(3);
+
+ if (la != null) {
+ final Map<String, Number> statsCpuLoadAverage = ImmutableMap.of(
+ "sysO/la/1", la[0],
+ "sysO/la/5", la[1],
+ "sysO/la/15", la[2]
+ );
+ for (Map.Entry<String, Number> entry : statsCpuLoadAverage.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class TcpStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+
+ InternetProtocolStats ipstats = os.getInternetProtocolStats();
+ InternetProtocolStats.TcpStats tcpv4 = ipstats.getTCPv4Stats();
+
+ if (tcpv4 != null) {
+ final Map<String, Long> stats = diff.to(
+ "tcpv4", ImmutableMap.<String, Long>builder()
+ .put("sysO/tcpv4/activeOpens", tcpv4.getConnectionsActive())
+ .put("sysO/tcpv4/passiveOpens", tcpv4.getConnectionsPassive())
+ .put("sysO/tcpv4/attemptFails", tcpv4.getConnectionFailures())
+ .put("sysO/tcpv4/estabResets", tcpv4.getConnectionsReset())
+ .put("sysO/tcpv4/in/segs", tcpv4.getSegmentsReceived())
+ .put("sysO/tcpv4/in/errs", tcpv4.getInErrors())
+ .put("sysO/tcpv4/out/segs", tcpv4.getSegmentsSent())
+ .put("sysO/tcpv4/out/rsts", tcpv4.getOutResets())
+ .put("sysO/tcpv4/retrans/segs", tcpv4.getSegmentsRetransmitted())
+ .build()
+ );
+ if (stats != null) {
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ InternetProtocolStats.TcpStats tcpv6 = ipstats.getTCPv6Stats();
+
+ if (tcpv6 != null) {
+ final Map<String, Long> stats = diff.to(
+ "tcpv6", ImmutableMap.<String, Long>builder()
+ .put("sysO/tcpv6/activeOpens", tcpv6.getConnectionsActive())
+ .put("sysO/tcpv6/passiveOpens", tcpv6.getConnectionsPassive())
+ .put("sysO/tcpv6/attemptFails", tcpv6.getConnectionFailures())
+ .put("sysO/tcpv6/estabResets", tcpv6.getConnectionsReset())
+ .put("sysO/tcpv6/in/segs", tcpv6.getSegmentsReceived())
+ .put("sysO/tcpv6/in/errs", tcpv6.getInErrors())
+ .put("sysO/tcpv6/out/segs", tcpv6.getSegmentsSent())
+ .put("sysO/tcpv6/out/rsts", tcpv6.getOutResets())
+ .put("sysO/tcpv6/retrans/segs", tcpv6.getSegmentsRetransmitted())
+ .build()
+ );
+ if (stats != null) {
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
Review Comment:
There is also a difference in how hardware address is being represented. E.g.
metric, Oshi value, sigar value
```
.sys/net/read/packets.userDims.netAddress,a4:83:e7:9b:de:d4,100.217.188.78
.sys/net/read/packets.userDims.netHwaddr,[2405:201:5806:a040:cc6b:688e:7502:3991, 2405:201:5806:a040:1490:85f8:f64f:42b4, fe80:0:0:0:1406:7632:fc08:2633],00:00:00:00:00:00
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1214266902
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(SysMonitorOshi.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public SysMonitorOshi()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/mem/max", mem.getTotal(),
Review Comment:
Sounds correct. Retained
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1214106306
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(SysMonitorOshi.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public SysMonitorOshi()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/mem/max", mem.getTotal(),
+ "sysO/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sysO/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/swap/pageIn", currPageIn - prevPageIn,
+ "sysO/swap/pageOut", currPageOut - prevPageOut,
+ "sysO/swap/max", swap.getSwapTotal(),
+ "sysO/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ if (fsTypeWhitelist.contains(fs.getType())) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sysO/fs/max", fs.getTotalSpace())
+ .put("sysO/fs/used", fs.getUsableSpace())
+ .put("sysO/fs/files/count", fs.getTotalInodes())
+ .put("sysO/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getName())
+ .setDimension("fsDirName", fs.getVolume())
+ .setDimension("fsTypeName", fs.getType())
+ .setDimension("fsSysTypeName", fs.getDescription())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sysO/disk/read/size", disk.getReads())
+ .put("sysO/disk/read/count", disk.getReadBytes())
+ .put("sysO/disk/write/size", disk.getWrites())
+ .put("sysO/disk/write/count", disk.getWriteBytes())
+ .put("sysO/disk/queue", disk.getCurrentQueueLength())
+ .put("sysO/disk/transferTime", disk.getTransferTime())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("diskName", disk.getName())
+ .setDimension("diskModel", disk.getModel());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class NetStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<NetworkIF> networkIFS = hal.getNetworkIFs();
+ for (NetworkIF net : networkIFS) {
+ final String name = net.getName();
+ for (String addr : net.getIPv4addr()) {
+ if (netAddressBlacklist.contains(addr)) {
+ net = null;
+ log.debug("Not monitoring network stats for ip addr: [%s], network name: [%s]", addr, name);
+ break;
+ }
+ }
+ if (net != null) {
+
+ final Map<String, Long> stats = diff.to(
+ name,
+ ImmutableMap.<String, Long>builder()
+ .put("sysO/net/read/size", net.getBytesRecv())
+ .put("sysO/net/read/packets", net.getPacketsRecv())
+ .put("sysO/net/read/errors", net.getInErrors())
+ .put("sysO/net/read/dropped", net.getInDrops())
+ .put("sysO/net/write/size", net.getBytesSent())
+ .put("sysO/net/write/packets", net.getPacketsSent())
+ .put("sysO/net/write/errors", net.getOutErrors())
+ .put("sysO/net/write/collisions", net.getCollisions())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("netName", net.getName())
+ .setDimension("netAddress", net.getMacaddr())
+ .setDimension("netHwaddr", Arrays.toString(net.getIPv6addr()));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+ }
+
+ private class CpuStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ CentralProcessor processor = hal.getProcessor();
+ long[][] procTicks = processor.getProcessorCpuLoadTicks();
+ for (int i = 0; i < procTicks.length; ++i) {
+ final String name = Integer.toString(i);
+ long[] ticks = procTicks[i];
+ long user = ticks[TickType.USER.getIndex()];
+ long nice = ticks[TickType.NICE.getIndex()];
+ long sys = ticks[TickType.SYSTEM.getIndex()];
+ long idle = ticks[TickType.IDLE.getIndex()];
+ long iowait = ticks[TickType.IOWAIT.getIndex()];
+ long irq = ticks[TickType.IRQ.getIndex()];
+ long softirq = ticks[TickType.SOFTIRQ.getIndex()];
+ long steal = ticks[TickType.STEAL.getIndex()];
+ long totalCpu = user + nice + sys + idle + iowait + irq + softirq + steal;
+ final Map<String, Long> stats = diff.to(
+ name,
+ ImmutableMap.<String, Long>builder()
+ .put("user", user) // user = Δuser / Δtotal
+ .put("sys", sys) // sys = Δsys / Δtotal
+ .put("nice", nice) // nice = Δnice / Δtotal
+ .put("wait", iowait) // wait = Δwait / Δtotal
+ .put("irq", irq) // irq = Δirq / Δtotal
+ .put("softIrq", softirq) // softIrq = ΔsoftIrq / Δtotal
+ .put("stolen", steal) // stolen = Δstolen / Δtotal
+ .put("_total", totalCpu) // (not reported)
+ .build()
+ );
+ if (stats != null) {
+ final long total = stats.remove("_total");
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("cpuName", name)
+ .setDimension("cpuTime", entry.getKey());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ if (total != 0) {
+ // prevent divide by 0 exception and don't emit such events
+ emitter.emit(builder.build("sysO/cpu", entry.getValue() * 100 / total)); // [0,100]
+ }
+
+ }
+ }
+ }
+ }
+ }
+
+ private class SysStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+
+ long uptime = os.getSystemUptime();
+
+ final Map<String, Number> stats = ImmutableMap.of(
+ "sysO/uptime", uptime
+ );
+ for (Map.Entry<String, Number> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ CentralProcessor processor = hal.getProcessor();
+ double[] la = processor.getSystemLoadAverage(3);
+
+ if (la != null) {
+ final Map<String, Number> statsCpuLoadAverage = ImmutableMap.of(
+ "sysO/la/1", la[0],
+ "sysO/la/5", la[1],
+ "sysO/la/15", la[2]
+ );
+ for (Map.Entry<String, Number> entry : statsCpuLoadAverage.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class TcpStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+
+ InternetProtocolStats ipstats = os.getInternetProtocolStats();
+ InternetProtocolStats.TcpStats tcpv4 = ipstats.getTCPv4Stats();
+
+ if (tcpv4 != null) {
+ final Map<String, Long> stats = diff.to(
+ "tcpv4", ImmutableMap.<String, Long>builder()
+ .put("sysO/tcpv4/activeOpens", tcpv4.getConnectionsActive())
+ .put("sysO/tcpv4/passiveOpens", tcpv4.getConnectionsPassive())
+ .put("sysO/tcpv4/attemptFails", tcpv4.getConnectionFailures())
+ .put("sysO/tcpv4/estabResets", tcpv4.getConnectionsReset())
+ .put("sysO/tcpv4/in/segs", tcpv4.getSegmentsReceived())
+ .put("sysO/tcpv4/in/errs", tcpv4.getInErrors())
+ .put("sysO/tcpv4/out/segs", tcpv4.getSegmentsSent())
+ .put("sysO/tcpv4/out/rsts", tcpv4.getOutResets())
+ .put("sysO/tcpv4/retrans/segs", tcpv4.getSegmentsRetransmitted())
+ .build()
+ );
+ if (stats != null) {
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ InternetProtocolStats.TcpStats tcpv6 = ipstats.getTCPv6Stats();
+
+ if (tcpv6 != null) {
+ final Map<String, Long> stats = diff.to(
+ "tcpv6", ImmutableMap.<String, Long>builder()
+ .put("sysO/tcpv6/activeOpens", tcpv6.getConnectionsActive())
+ .put("sysO/tcpv6/passiveOpens", tcpv6.getConnectionsPassive())
+ .put("sysO/tcpv6/attemptFails", tcpv6.getConnectionFailures())
+ .put("sysO/tcpv6/estabResets", tcpv6.getConnectionsReset())
+ .put("sysO/tcpv6/in/segs", tcpv6.getSegmentsReceived())
+ .put("sysO/tcpv6/in/errs", tcpv6.getInErrors())
+ .put("sysO/tcpv6/out/segs", tcpv6.getSegmentsSent())
+ .put("sysO/tcpv6/out/rsts", tcpv6.getOutResets())
+ .put("sysO/tcpv6/retrans/segs", tcpv6.getSegmentsRetransmitted())
+ .build()
+ );
+ if (stats != null) {
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
Review Comment:
>We can leave v6 metrics out for now.
Should I add tcpv4 and tcpv6 to give tcp metrics?
> We also have these metrics missing in Oshi
Yeah these metrics are missing in Oshi implementation. I'll share you output of every metrics that Oshi can release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1220854096
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", type)
+ .setDimension("fsSysTypeName", fs.getType())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ // Difference b/w metrics of two consecutive values. It tells Δmetric (increase/decrease in metrics value)
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+
+ final Map<String, Long> stats = diff.to(
+ disk.getName(),
+ ImmutableMap.<String, Long>builder()
+ .put("sys/disk/read/size", disk.getReads())
+ .put("sys/disk/read/count", disk.getReadBytes())
+ .put("sys/disk/write/size", disk.getWrites())
+ .put("sys/disk/write/count", disk.getWriteBytes())
+ .put("sys/disk/queue", disk.getCurrentQueueLength())
+ .put("sys/disk/transferTime", disk.getTransferTime())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("diskName", disk.getName())
+ .setDimension("diskModel", disk.getModel());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+
+ private class NetStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<NetworkIF> networkIFS = hal.getNetworkIFs();
+ for (NetworkIF net : networkIFS) {
+ final String name = net.getName();
+ for (String addr : net.getIPv4addr()) {
+ if (netAddressBlacklist.contains(addr)) {
+ net = null;
+ log.debug("Not monitoring network stats for ip addr: [%s], network name: [%s]", addr, name);
+ break;
+ }
+ }
+ if (net != null) {
+
+ final Map<String, Long> stats = diff.to(
+ name,
+ ImmutableMap.<String, Long>builder()
+ .put("sys/net/read/size", net.getBytesRecv())
+ .put("sys/net/read/packets", net.getPacketsRecv())
+ .put("sys/net/read/errors", net.getInErrors())
+ .put("sys/net/read/dropped", net.getInDrops())
+ .put("sys/net/write/size", net.getBytesSent())
+ .put("sys/net/write/packets", net.getPacketsSent())
+ .put("sys/net/write/errors", net.getOutErrors())
+ .put("sys/net/write/collisions", net.getCollisions())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("netName", net.getName())
+ .setDimension("netAddress", Arrays.toString(net.getIPv4addr()))
+ .setDimension("netHwaddr", net.getMacaddr());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+ }
+
+ private class CpuStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ CentralProcessor processor = hal.getProcessor();
+ long[][] procTicks = processor.getProcessorCpuLoadTicks();
+ for (int i = 0; i < procTicks.length; ++i) {
+ final String name = Integer.toString(i);
+ long[] ticks = procTicks[i];
+ long user = ticks[TickType.USER.getIndex()];
+ long nice = ticks[TickType.NICE.getIndex()];
+ long sys = ticks[TickType.SYSTEM.getIndex()];
+ long idle = ticks[TickType.IDLE.getIndex()];
+ long iowait = ticks[TickType.IOWAIT.getIndex()];
+ long irq = ticks[TickType.IRQ.getIndex()];
+ long softirq = ticks[TickType.SOFTIRQ.getIndex()];
+ long steal = ticks[TickType.STEAL.getIndex()];
+ long totalCpu = user + nice + sys + idle + iowait + irq + softirq + steal;
Review Comment:
yes, confirmed from both Oshi SystemInfo test file and how it is implemented in SysMonitor.
Let me know if something looks off to you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1220854096
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", type)
+ .setDimension("fsSysTypeName", fs.getType())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ // Difference b/w metrics of two consecutive values. It tells Δmetric (increase/decrease in metrics value)
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+
+ final Map<String, Long> stats = diff.to(
+ disk.getName(),
+ ImmutableMap.<String, Long>builder()
+ .put("sys/disk/read/size", disk.getReads())
+ .put("sys/disk/read/count", disk.getReadBytes())
+ .put("sys/disk/write/size", disk.getWrites())
+ .put("sys/disk/write/count", disk.getWriteBytes())
+ .put("sys/disk/queue", disk.getCurrentQueueLength())
+ .put("sys/disk/transferTime", disk.getTransferTime())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("diskName", disk.getName())
+ .setDimension("diskModel", disk.getModel());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+
+ private class NetStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<NetworkIF> networkIFS = hal.getNetworkIFs();
+ for (NetworkIF net : networkIFS) {
+ final String name = net.getName();
+ for (String addr : net.getIPv4addr()) {
+ if (netAddressBlacklist.contains(addr)) {
+ net = null;
+ log.debug("Not monitoring network stats for ip addr: [%s], network name: [%s]", addr, name);
+ break;
+ }
+ }
+ if (net != null) {
+
+ final Map<String, Long> stats = diff.to(
+ name,
+ ImmutableMap.<String, Long>builder()
+ .put("sys/net/read/size", net.getBytesRecv())
+ .put("sys/net/read/packets", net.getPacketsRecv())
+ .put("sys/net/read/errors", net.getInErrors())
+ .put("sys/net/read/dropped", net.getInDrops())
+ .put("sys/net/write/size", net.getBytesSent())
+ .put("sys/net/write/packets", net.getPacketsSent())
+ .put("sys/net/write/errors", net.getOutErrors())
+ .put("sys/net/write/collisions", net.getCollisions())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("netName", net.getName())
+ .setDimension("netAddress", Arrays.toString(net.getIPv4addr()))
+ .setDimension("netHwaddr", net.getMacaddr());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+ }
+
+ private class CpuStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ CentralProcessor processor = hal.getProcessor();
+ long[][] procTicks = processor.getProcessorCpuLoadTicks();
+ for (int i = 0; i < procTicks.length; ++i) {
+ final String name = Integer.toString(i);
+ long[] ticks = procTicks[i];
+ long user = ticks[TickType.USER.getIndex()];
+ long nice = ticks[TickType.NICE.getIndex()];
+ long sys = ticks[TickType.SYSTEM.getIndex()];
+ long idle = ticks[TickType.IDLE.getIndex()];
+ long iowait = ticks[TickType.IOWAIT.getIndex()];
+ long irq = ticks[TickType.IRQ.getIndex()];
+ long softirq = ticks[TickType.SOFTIRQ.getIndex()];
+ long steal = ticks[TickType.STEAL.getIndex()];
+ long totalCpu = user + nice + sys + idle + iowait + irq + softirq + steal;
Review Comment:
yes, confirmed from both Oshi SystemInfo test file and how it is implemented in SysMonitor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1219491221
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
Review Comment:
is it required?
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
Review Comment:
please add javadocs like there are in SysMonitor
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
Review Comment:
this can be simplified. There is no need of fsTypeWhitelist. if the type is known in localFsSysTypes, we emit the metrics and not otherwise.
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", type)
+ .setDimension("fsSysTypeName", fs.getType())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ // Difference b/w metrics of two consecutive values. It tells Δmetric (increase/decrease in metrics value)
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+
+ final Map<String, Long> stats = diff.to(
+ disk.getName(),
+ ImmutableMap.<String, Long>builder()
+ .put("sys/disk/read/size", disk.getReads())
+ .put("sys/disk/read/count", disk.getReadBytes())
+ .put("sys/disk/write/size", disk.getWrites())
+ .put("sys/disk/write/count", disk.getWriteBytes())
+ .put("sys/disk/queue", disk.getCurrentQueueLength())
+ .put("sys/disk/transferTime", disk.getTransferTime())
Review Comment:
does't look right to me. I would assume that *Writebytes represent size and writes represent count. Ditto for reads.
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", type)
+ .setDimension("fsSysTypeName", fs.getType())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ // Difference b/w metrics of two consecutive values. It tells Δmetric (increase/decrease in metrics value)
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+
+ final Map<String, Long> stats = diff.to(
+ disk.getName(),
+ ImmutableMap.<String, Long>builder()
+ .put("sys/disk/read/size", disk.getReads())
+ .put("sys/disk/read/count", disk.getReadBytes())
+ .put("sys/disk/write/size", disk.getWrites())
+ .put("sys/disk/write/count", disk.getWriteBytes())
+ .put("sys/disk/queue", disk.getCurrentQueueLength())
+ .put("sys/disk/transferTime", disk.getTransferTime())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("diskName", disk.getName())
+ .setDimension("diskModel", disk.getModel());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+
+ private class NetStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<NetworkIF> networkIFS = hal.getNetworkIFs();
+ for (NetworkIF net : networkIFS) {
+ final String name = net.getName();
+ for (String addr : net.getIPv4addr()) {
Review Comment:
is it supposed to return multiple addresses?
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
Review Comment:
lets initialize in constructor.
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
Review Comment:
this can be static.
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
Review Comment:
can you add comments as to what "used" actually refers to here? That is does it include cache/buffers etc? Or you can add that in the docs
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
Review Comment:
usable space is different from used space.
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", type)
+ .setDimension("fsSysTypeName", fs.getType())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
Review Comment:
the implementation is different from Sigar. Apologies if you told me and I don't remember.
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", type)
+ .setDimension("fsSysTypeName", fs.getType())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ // Difference b/w metrics of two consecutive values. It tells Δmetric (increase/decrease in metrics value)
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+
+ final Map<String, Long> stats = diff.to(
+ disk.getName(),
+ ImmutableMap.<String, Long>builder()
+ .put("sys/disk/read/size", disk.getReads())
+ .put("sys/disk/read/count", disk.getReadBytes())
+ .put("sys/disk/write/size", disk.getWrites())
+ .put("sys/disk/write/count", disk.getWriteBytes())
+ .put("sys/disk/queue", disk.getCurrentQueueLength())
+ .put("sys/disk/transferTime", disk.getTransferTime())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("diskName", disk.getName())
+ .setDimension("diskModel", disk.getModel());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+
+ private class NetStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<NetworkIF> networkIFS = hal.getNetworkIFs();
+ for (NetworkIF net : networkIFS) {
+ final String name = net.getName();
+ for (String addr : net.getIPv4addr()) {
+ if (netAddressBlacklist.contains(addr)) {
+ net = null;
+ log.debug("Not monitoring network stats for ip addr: [%s], network name: [%s]", addr, name);
+ break;
+ }
+ }
+ if (net != null) {
+
+ final Map<String, Long> stats = diff.to(
+ name,
+ ImmutableMap.<String, Long>builder()
+ .put("sys/net/read/size", net.getBytesRecv())
+ .put("sys/net/read/packets", net.getPacketsRecv())
+ .put("sys/net/read/errors", net.getInErrors())
+ .put("sys/net/read/dropped", net.getInDrops())
+ .put("sys/net/write/size", net.getBytesSent())
+ .put("sys/net/write/packets", net.getPacketsSent())
+ .put("sys/net/write/errors", net.getOutErrors())
+ .put("sys/net/write/collisions", net.getCollisions())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("netName", net.getName())
+ .setDimension("netAddress", Arrays.toString(net.getIPv4addr()))
+ .setDimension("netHwaddr", net.getMacaddr());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+ }
+
+ private class CpuStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ CentralProcessor processor = hal.getProcessor();
+ long[][] procTicks = processor.getProcessorCpuLoadTicks();
+ for (int i = 0; i < procTicks.length; ++i) {
+ final String name = Integer.toString(i);
+ long[] ticks = procTicks[i];
+ long user = ticks[TickType.USER.getIndex()];
+ long nice = ticks[TickType.NICE.getIndex()];
+ long sys = ticks[TickType.SYSTEM.getIndex()];
+ long idle = ticks[TickType.IDLE.getIndex()];
+ long iowait = ticks[TickType.IOWAIT.getIndex()];
+ long irq = ticks[TickType.IRQ.getIndex()];
+ long softirq = ticks[TickType.SOFTIRQ.getIndex()];
+ long steal = ticks[TickType.STEAL.getIndex()];
+ long totalCpu = user + nice + sys + idle + iowait + irq + softirq + steal;
Review Comment:
is this the correct definition?
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SysMonitor implemented using {@link oshi}
+ * <p>
+ * Following stats are emitted:
+ * <ul>
+ * <li>{@link MemStats} for Memory related metrics</li>
+ * <li>{@link SwapStats} for swap storage related metrics</li>
+ * <li>{@link FsStats} for File System related Metrics</li>
+ * <li>{@link DiskStats} for Disk level metrics</li>
+ * <li>{@link NetStats} for Network Interface and related metrics</li>
+ * <li>{@link CpuStats} for CPU usage and stats metrics</li>
+ * <li>{@link SysStats} for overall system metrics(uptime, avg load)</li>
+ * <li>{@link TcpStats} for TCP related metrics</li>
+ * </ul>
+ */
+public class OshiSysMonitor extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(OshiSysMonitor.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<String> localFsSysTypes = ImmutableList.of(
+ "apfs",
+ "cvfs",
+ "msdos",
+ "minix",
+ "hpfs",
+ "vxfs",
+ "vfat",
+ "vfat",
+ "zfs",
+ "lifs"
+ );
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public OshiSysMonitor()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public OshiSysMonitor(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ /**
+ * Interface to implement Stats
+ * <p>
+ * Define a method {@link #emit(ServiceEmitter)} to emit metrices in emiters
+ */
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/mem/max", mem.getTotal(),
+ "sys/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sys/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sys/swap/pageIn", currPageIn - prevPageIn,
+ "sys/swap/pageOut", currPageOut - prevPageOut,
+ "sys/swap/max", swap.getSwapTotal(),
+ "sys/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ final String type = localFsSysTypes.contains(fs.getType()) ? "local" : "unknown";
+ if (fsTypeWhitelist.contains(type)) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sys/fs/max", fs.getTotalSpace())
+ .put("sys/fs/used", fs.getUsableSpace())
+ .put("sys/fs/files/count", fs.getTotalInodes())
+ .put("sys/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getVolume())
+ .setDimension("fsDirName", fs.getMount())
+ .setDimension("fsTypeName", type)
+ .setDimension("fsSysTypeName", fs.getType())
+ .setDimension("fsOptions", fs.getOptions().split(","));
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ } else {
+ log.debug("Not monitoring fs stats for name[%s] with typeName[%s]", name, fs.getType());
+ }
+ }
+ }
+ }
+
+ private class DiskStats implements Stats
+ {
+ // Difference b/w metrics of two consecutive values. It tells Δmetric (increase/decrease in metrics value)
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<HWDiskStore> disks = hal.getDiskStores();
+ // disk partitions can be mapped to file system but no inbuilt method is there to find relation b/w disks and file system
+ // Will have to add logic for that
+ for (HWDiskStore disk : disks) {
+
+ final Map<String, Long> stats = diff.to(
+ disk.getName(),
+ ImmutableMap.<String, Long>builder()
+ .put("sys/disk/read/size", disk.getReads())
+ .put("sys/disk/read/count", disk.getReadBytes())
+ .put("sys/disk/write/size", disk.getWrites())
+ .put("sys/disk/write/count", disk.getWriteBytes())
+ .put("sys/disk/queue", disk.getCurrentQueueLength())
+ .put("sys/disk/transferTime", disk.getTransferTime())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("diskName", disk.getName())
+ .setDimension("diskModel", disk.getModel());
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+ }
+
+ private class NetStats implements Stats
+ {
+ private final KeyedDiff diff = new KeyedDiff();
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ List<NetworkIF> networkIFS = hal.getNetworkIFs();
+ for (NetworkIF net : networkIFS) {
+ final String name = net.getName();
+ for (String addr : net.getIPv4addr()) {
+ if (netAddressBlacklist.contains(addr)) {
+ net = null;
+ log.debug("Not monitoring network stats for ip addr: [%s], network name: [%s]", addr, name);
+ break;
+ }
+ }
+ if (net != null) {
+
+ final Map<String, Long> stats = diff.to(
+ name,
+ ImmutableMap.<String, Long>builder()
+ .put("sys/net/read/size", net.getBytesRecv())
+ .put("sys/net/read/packets", net.getPacketsRecv())
+ .put("sys/net/read/errors", net.getInErrors())
+ .put("sys/net/read/dropped", net.getInDrops())
+ .put("sys/net/write/size", net.getBytesSent())
+ .put("sys/net/write/packets", net.getPacketsSent())
+ .put("sys/net/write/errors", net.getOutErrors())
+ .put("sys/net/write/collisions", net.getCollisions())
+ .build()
+ );
+ if (stats != null) {
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("netName", net.getName())
+ .setDimension("netAddress", Arrays.toString(net.getIPv4addr()))
Review Comment:
this doesn't seem right. lets discuss offline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1234724080
##########
processing/src/main/java/org/apache/druid/java/util/metrics/OshiSysMonitor.java:
##########
@@ -313,8 +313,9 @@ public void emit(ServiceEmitter emitter)
for (String addr : net.getIPv4addr()) {
if (!NET_ADDRESS_BLACKLIST.contains(addr)) {
// Only emit metrics for non black-listed ip addresses
+ String mapKey = name + "_" + addr; // Network_Name_IPV4 address as key, ex: wifi_192.1.0.1 to uniquely identify the dimension
Review Comment:
Yes they were 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1213948573
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(SysMonitorOshi.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public SysMonitorOshi()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/mem/max", mem.getTotal(),
+ "sysO/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sysO/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/swap/pageIn", currPageIn - prevPageIn,
+ "sysO/swap/pageOut", currPageOut - prevPageOut,
+ "sysO/swap/max", swap.getSwapTotal(),
+ "sysO/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ if (fsTypeWhitelist.contains(fs.getType())) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sysO/fs/max", fs.getTotalSpace())
+ .put("sysO/fs/used", fs.getUsableSpace())
+ .put("sysO/fs/files/count", fs.getTotalInodes())
+ .put("sysO/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getName())
Review Comment:
I think `fs.getName()` returns volume name only. We're emitting 160 metrics in OshiSysMonitor, Let me share them on slack and you can verify if there's some issue. That would be easier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on PR #14359:
URL: https://github.com/apache/druid/pull/14359#issuecomment-1599011681
Thanks @abhishekagarwal87 for giving me this great learning opportunity, in form of a very meaningful Good First issue :). Also thanks for being super supportive and helping me out with your reviews, and teachings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1213404836
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
Review Comment:
can you add some javadocs describing this class? Also a nit, we can use `Oshi` as prefix instead of suffix.
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
Review Comment:
can you also deprecate `SysMonitor` class and add a note in that class that OshiSysMonitor is supposed to be used and maintained.
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(SysMonitorOshi.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public SysMonitorOshi()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/mem/max", mem.getTotal(),
+ "sysO/mem/used", mem.getTotal() - mem.getAvailable(),
+ "sysO/mem/free", mem.getAvailable()
+ );
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+ }
+ }
+ }
+
+ private class SwapStats implements Stats
+ {
+ private long prevPageIn = 0;
+ private long prevPageOut = 0;
+
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ VirtualMemory swap = hal.getMemory().getVirtualMemory();
+
+ if (swap != null) {
+ long currPageIn = swap.getSwapPagesIn();
+ long currPageOut = swap.getSwapPagesOut();
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/swap/pageIn", currPageIn - prevPageIn,
+ "sysO/swap/pageOut", currPageOut - prevPageOut,
+ "sysO/swap/max", swap.getSwapTotal(),
+ "sysO/swap/free", swap.getSwapTotal() - swap.getSwapUsed()
+ );
+
+ final ServiceMetricEvent.Builder builder = builder();
+ MonitorUtils.addDimensionsToBuilder(builder, dimensions);
+ for (Map.Entry<String, Long> entry : stats.entrySet()) {
+ emitter.emit(builder.build(entry.getKey(), entry.getValue()));
+ }
+
+ this.prevPageIn = currPageIn;
+ this.prevPageOut = currPageOut;
+ }
+ }
+ }
+
+ private class FsStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ FileSystem fileSystem = os.getFileSystem();
+ for (OSFileStore fs : fileSystem.getFileStores()) {
+ final String name = fs.getName();
+ if (fsTypeWhitelist.contains(fs.getType())) {
+ final Map<String, Long> stats = ImmutableMap.<String, Long>builder()
+ .put("sysO/fs/max", fs.getTotalSpace())
+ .put("sysO/fs/used", fs.getUsableSpace())
+ .put("sysO/fs/files/count", fs.getTotalInodes())
+ .put("sysO/fs/files/free", fs.getFreeInodes())
+ .build();
+ final ServiceMetricEvent.Builder builder = builder()
+ .setDimension("fsDevName", fs.getName())
Review Comment:
I think that volume name should be used for fsDevName and Name for the dirName dimension. thoughts?
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
+{
+ private static final Logger log = new Logger(SysMonitorOshi.class);
+ private final SystemInfo si = new SystemInfo();
+ private final HardwareAbstractionLayer hal = si.getHardware();
+ private final OperatingSystem os = si.getOperatingSystem();
+ private final List<String> fsTypeWhitelist = ImmutableList.of("local");
+ private final List<String> netAddressBlacklist = ImmutableList.of("0.0.0.0", "127.0.0.1");
+ private final List<Stats> statsList;
+
+ private final Map<String, String[]> dimensions;
+
+ public SysMonitorOshi()
+ {
+ this(ImmutableMap.of());
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions)
+ {
+ this(dimensions, DEFAULT_METRICS_FEED);
+ }
+
+ public SysMonitorOshi(Map<String, String[]> dimensions, String feed)
+ {
+ super(feed);
+ Preconditions.checkNotNull(dimensions);
+ this.dimensions = ImmutableMap.copyOf(dimensions);
+
+ this.statsList = new ArrayList<Stats>();
+ this.statsList.addAll(
+ Arrays.asList(
+ new MemStats(),
+ new SwapStats(),
+ new FsStats(),
+ new DiskStats(),
+ new NetStats(),
+ new CpuStats(),
+ new SysStats(),
+ new TcpStats()
+ )
+ );
+ }
+
+ @Override
+ public boolean doMonitor(ServiceEmitter emitter)
+ {
+ for (Stats stats : statsList) {
+ stats.emit(emitter);
+ }
+ return true;
+ }
+
+ private interface Stats
+ {
+ void emit(ServiceEmitter emitter);
+ }
+
+ private class MemStats implements Stats
+ {
+ @Override
+ public void emit(ServiceEmitter emitter)
+ {
+ GlobalMemory mem = hal.getMemory();
+ if (mem != null) {
+ final Map<String, Long> stats = ImmutableMap.of(
+ "sysO/mem/max", mem.getTotal(),
Review Comment:
the metric names need not change for this monitor. We expect only one of the monitor will be active at a time.
##########
processing/src/test/java/org/apache/druid/java/util/metrics/SysMonitorOshiTest.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.emitter.core.Event;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+import oshi.util.Util;
+
+import java.util.List;
+
+public class SysMonitorOshiTest
Review Comment:
We should add more tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] hardikbajaj commented on a diff in pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "hardikbajaj (via GitHub)" <gi...@apache.org>.
hardikbajaj commented on code in PR #14359:
URL: https://github.com/apache/druid/pull/14359#discussion_r1213946433
##########
processing/src/main/java/org/apache/druid/java/util/metrics/SysMonitorOshi.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.metrics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import oshi.SystemInfo;
+import oshi.hardware.CentralProcessor;
+import oshi.hardware.CentralProcessor.TickType;
+import oshi.hardware.GlobalMemory;
+import oshi.hardware.HWDiskStore;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.hardware.NetworkIF;
+import oshi.hardware.VirtualMemory;
+import oshi.software.os.FileSystem;
+import oshi.software.os.InternetProtocolStats;
+import oshi.software.os.OSFileStore;
+import oshi.software.os.OperatingSystem;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+
+public class SysMonitorOshi extends FeedDefiningMonitor
Review Comment:
Sure I'll do that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on pull request #14359: Added new SysMonitorOshi v0 using Oshi library
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on PR #14359:
URL: https://github.com/apache/druid/pull/14359#issuecomment-1573177589
the 'check-licenses' script we have to help us ensure that we collect all the correct licensing information for our source distributions to meet apache requirements [has entries for the licenses it is aware of to normalize them into standard buckets](https://github.com/apache/druid/blob/master/distribution/bin/check-licenses.py#L294), it looks like maybe something is reporting a string that it hasn't seen before, which looks like what this error is perhaps:
```
KeyError: 'SPDX-License-Identifier: MIT'
```
I think if you add that entry into the script and bucket it into the other MIT licenses then it should probably be ok?
Also if there are any transitive dependencies pulled in by this new library they might also need added to the licenses.yaml file. I know this stuff is somewhat tedious, so apologies for that, and still not a perfect system by any means because it is prone to becoming stale, but it does significantly reduce the burden of doing releases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org