You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/05/19 13:31:32 UTC
[GitHub] [ozone] ArafatKhan2198 opened a new pull request, #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
ArafatKhan2198 opened a new pull request, #3430:
URL: https://github.com/apache/ozone/pull/3430
## What changes were proposed in this pull request?
Add metrics for volume statistics including disk capacity, usage, type (SSD/HDD).
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6705
## How was this patch tested?
The Metrics have been exposed onto the JMX of the DataNode
```
{
"name" : "Hadoop:service=HddsDatanode,name=VolumeInfoStats-/tmp/hadoop-arafat2198/dfs/data",
"modelerType" : "VolumeInfoStats-/tmp/hadoop-arafat2198/dfs/data",
"tag.Context" : "ozone",
"tag.Hostname" : "arafat2198-Aspire-A315-51",
"Capacity" : 425151766528,
"SpaceAvailable" : 425151766528,
"SpaceReserved" : 0,
"SpaceUsed" : 0,
"TotalCapacity" : 425151766528
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r878007261
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableGaugeLong;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume IO stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics", context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String VolumeRootStr;
+ private HddsVolume volume;
+
+ private @Metric MutableGaugeLong spaceUsed;
+ private @Metric MutableGaugeLong spaceAvailable;
+ private @Metric MutableGaugeLong spaceReserved;
+ private @Metric MutableGaugeLong capacity;
+ private @Metric MutableGaugeLong totalCapacity;
+
+
+ @Deprecated
Review Comment:
Thanks! I have changed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jojochuang merged pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
jojochuang merged PR #3430:
URL: https://github.com/apache/ozone/pull/3430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r883075680
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume Info stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics",
+ context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String volumeRootStr;
+ private HddsVolume volume;
+
+ private long spaceUsed;
+ private long spaceAvailable;
+ private long spaceReserved;
+
+
+ /**
+ * @param identifier Typically, path to volume root. e.g. /data/hdds
+ */
+ public VolumeInfoStats(String identifier, HddsVolume ref) {
+ this.metricsSourceName +=
+ '-' + identifier + StorageVolume.VolumeType.DATA_VOLUME.toString();
+ this.volumeRootStr = identifier;
+ this.volume = ref;
+ init();
+ }
+
+ public void init() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.register(metricsSourceName, "Volume Info Statistics", this);
+ }
+
+ public void unregister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(metricsSourceName);
+ }
+
+ @Metric("Metric to return the Storage Type")
+ public String getStorageType() {
+ return volume.getStorageType().toString();
+ }
+
+ @Metric("Returns the Directory name for the volume")
+ public String getStorageDirectory() {
+ return volume.getStorageDir().toString();
+ }
+
+ @Metric("Return the DataNode UID for the respective volume")
+ public String getDatanodeUuid() {
+ return volume.getDatanodeUuid();
+ }
+
+ @Metric("Return the Layout Version for the volume")
+ public int getLayoutVersion() {
+ return volume.getLayoutVersion();
+ }
+
+ @Metric("Returns the Volume Type")
+ public String getVolumeType() {
+ return volume.getType().name();
+ }
+
+ public String getMetricsSourceName() {
+ return metricsSourceName;
+ }
+
+ /**
+ * Test conservative avail space.
+ * |----used----| (avail) |++++++++reserved++++++++|
+ * |<------- capacity ------->|
+ * |<------------------- Total capacity -------------->|
+ * A) avail = capacity - used
+ * B) capacity = used + avail
+ * C) Total capacity = used + avail + reserved
+ */
+
+ /**
+ * Return the Storage type for the Volume.
+ */
+ @Metric("Returns the Used space")
+ public long getUsed() {
+ spaceUsed = volume.getVolumeInfo().getScmUsed();
+ return spaceUsed;
+ }
+
+ /**
+ * Return the Total Available capacity of the Volume.
+ */
+ @Metric("Returns the Available space")
+ public long getAvailable() {
+ spaceAvailable = volume.getVolumeInfo().getAvailable();
+ return spaceAvailable;
+ }
+
+ /**
+ * Return the Total Reserved of the Volume.
+ */
+ @Metric("Fetches the Reserved Space")
+ public long getReserved() {
+ spaceReserved = volume.getVolumeInfo().getReservedInBytes();
+ return spaceReserved;
+ }
+
+ /**
+ * Return the Total capacity of the Volume.
+ */
+ @Metric("Returns the Capacity of the Volume")
+ public long getCapacity() {
+ return spaceUsed + spaceAvailable;
+ }
+
+ /**
+ * Return the Total capacity of the Volume.
+ */
+ @Metric("Returns the Total Capacity of the Volume")
+ public long getTotalCapacity() {
+ return (spaceUsed + spaceAvailable + spaceReserved);
Review Comment:
That's a good catch! will do!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r878007107
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableGaugeLong;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume IO stats for each HDDS Volume.
Review Comment:
Thanks! I have changed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r877651130
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableGaugeLong;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume IO stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics", context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String VolumeRootStr;
+ private HddsVolume volume;
+
+ private @Metric MutableGaugeLong spaceUsed;
+ private @Metric MutableGaugeLong spaceAvailable;
+ private @Metric MutableGaugeLong spaceReserved;
+ private @Metric MutableGaugeLong capacity;
+ private @Metric MutableGaugeLong totalCapacity;
+
+
+ @Deprecated
+ public VolumeInfoStats() {
+ init();
+ }
+
+ /**
+ * @param identifier Typically, path to volume root. e.g. /data/hdds
+ */
+ public VolumeInfoStats(String identifier, HddsVolume ref) {
+ this.metricsSourceName += '-' + identifier;
+ this.VolumeRootStr = identifier;
+ this.volume = ref;
+ init();
+ }
+
+ public void init() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.register(metricsSourceName, "Volume Info Statistics", this);
+ spaceUsed.set(volume.getVolumeInfo().getScmUsed());
+ spaceAvailable.set(volume.getVolumeInfo().getAvailable());
+ spaceReserved.set(volume.getVolumeInfo().getReservedInBytes());
+ capacity.set(spaceUsed.value() + spaceAvailable.value());
+ totalCapacity.set(
+ spaceUsed.value() + spaceAvailable.value() + spaceReserved.value());
+ }
+
+ public void unregister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(metricsSourceName);
+ }
+
+ public String getMetricsSourceName() {
+ return metricsSourceName;
+ }
+
+ /**
+ * Test conservative avail space.
+ * |----used----| (avail) |++++++++reserved++++++++|
+ * |<------- capacity ------->|
+ * |<------------------- Total capacity -------------->|
+ * A) avail = capacity - used
+ * B) avail = fsAvail - Max(reserved - other, 0);
+ */
Review Comment:
This comment seems out of place. The diagram of how space is broken up is quite useful though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jojochuang commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
jojochuang commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r877542797
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableGaugeLong;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume IO stats for each HDDS Volume.
Review Comment:
Not IO stats, but more like volume usage
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableGaugeLong;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume IO stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics", context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String VolumeRootStr;
+ private HddsVolume volume;
+
+ private @Metric MutableGaugeLong spaceUsed;
+ private @Metric MutableGaugeLong spaceAvailable;
+ private @Metric MutableGaugeLong spaceReserved;
+ private @Metric MutableGaugeLong capacity;
+ private @Metric MutableGaugeLong totalCapacity;
+
+
+ @Deprecated
+ public VolumeInfoStats() {
+ init();
+ }
+
+ /**
+ * @param identifier Typically, path to volume root. e.g. /data/hdds
+ */
+ public VolumeInfoStats(String identifier, HddsVolume ref) {
+ this.metricsSourceName += '-' + identifier;
+ this.VolumeRootStr = identifier;
+ this.volume = ref;
+ init();
+ }
+
+ public void init() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.register(metricsSourceName, "Volume Info Statistics", this);
+ spaceUsed.set(volume.getVolumeInfo().getScmUsed());
+ spaceAvailable.set(volume.getVolumeInfo().getAvailable());
+ spaceReserved.set(volume.getVolumeInfo().getReservedInBytes());
+ capacity.set(spaceUsed.value() + spaceAvailable.value());
+ totalCapacity.set(
+ spaceUsed.value() + spaceAvailable.value() + spaceReserved.value());
Review Comment:
```suggestion
```
these lines doesn't look needed.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableGaugeLong;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume IO stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics", context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String VolumeRootStr;
+ private HddsVolume volume;
+
+ private @Metric MutableGaugeLong spaceUsed;
+ private @Metric MutableGaugeLong spaceAvailable;
+ private @Metric MutableGaugeLong spaceReserved;
+ private @Metric MutableGaugeLong capacity;
+ private @Metric MutableGaugeLong totalCapacity;
+
+
+ @Deprecated
Review Comment:
why mark it deprecated? if users are not expected to call the constructor without parameters then remove it, or make it private?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r887270903
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java:
##########
@@ -332,19 +352,22 @@ public void shutdown() {
if (volumeIOStats != null) {
volumeIOStats.unregister();
}
+ if (volumeInfoMetrics != null) {
+ volumeInfoMetrics.unregister();
+ }
}
/**
* VolumeState represents the different states a HddsVolume can be in.
* NORMAL => Volume can be used for storage
* FAILED => Volume has failed due and can no longer be used for
- * storing containers.
Review Comment:
Yes I am sorry about that, Ill make sure it does not happen in the future!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r877651370
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableGaugeLong;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume IO stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics", context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String VolumeRootStr;
+ private HddsVolume volume;
+
+ private @Metric MutableGaugeLong spaceUsed;
+ private @Metric MutableGaugeLong spaceAvailable;
+ private @Metric MutableGaugeLong spaceReserved;
+ private @Metric MutableGaugeLong capacity;
+ private @Metric MutableGaugeLong totalCapacity;
+
+
+ @Deprecated
+ public VolumeInfoStats() {
+ init();
+ }
+
+ /**
+ * @param identifier Typically, path to volume root. e.g. /data/hdds
+ */
+ public VolumeInfoStats(String identifier, HddsVolume ref) {
+ this.metricsSourceName += '-' + identifier;
Review Comment:
Also, append the type of volume if it is `META_VOLUME` or `DATA_VOLUME`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r883075392
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume Info stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics",
+ context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
Review Comment:
Will do thanks !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r880080539
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableGaugeLong;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume IO stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics", context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String VolumeRootStr;
+ private HddsVolume volume;
+
+ private @Metric MutableGaugeLong spaceUsed;
+ private @Metric MutableGaugeLong spaceAvailable;
+ private @Metric MutableGaugeLong spaceReserved;
+ private @Metric MutableGaugeLong capacity;
+ private @Metric MutableGaugeLong totalCapacity;
+
+
+ @Deprecated
+ public VolumeInfoStats() {
+ init();
+ }
+
+ /**
+ * @param identifier Typically, path to volume root. e.g. /data/hdds
+ */
+ public VolumeInfoStats(String identifier, HddsVolume ref) {
+ this.metricsSourceName += '-' + identifier;
Review Comment:
Thanks for the comment
I have added a separate metric called VolumeType that deals with the type of volume !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jojochuang commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
jojochuang commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r883054420
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume Info stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics",
+ context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
Review Comment:
Rename as VolumeInfoMetrics
You can do this easily in IntelliJ: mouse right click on the class name, select "refactor" -> select "rename..."
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume Info stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics",
+ context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String volumeRootStr;
+ private HddsVolume volume;
+
+ private long spaceUsed;
+ private long spaceAvailable;
+ private long spaceReserved;
+
+
+ /**
+ * @param identifier Typically, path to volume root. e.g. /data/hdds
+ */
+ public VolumeInfoStats(String identifier, HddsVolume ref) {
+ this.metricsSourceName +=
+ '-' + identifier + StorageVolume.VolumeType.DATA_VOLUME.toString();
+ this.volumeRootStr = identifier;
+ this.volume = ref;
+ init();
+ }
+
+ public void init() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.register(metricsSourceName, "Volume Info Statistics", this);
+ }
+
+ public void unregister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(metricsSourceName);
+ }
+
+ @Metric("Metric to return the Storage Type")
+ public String getStorageType() {
+ return volume.getStorageType().toString();
+ }
+
+ @Metric("Returns the Directory name for the volume")
+ public String getStorageDirectory() {
+ return volume.getStorageDir().toString();
+ }
+
+ @Metric("Return the DataNode UID for the respective volume")
+ public String getDatanodeUuid() {
+ return volume.getDatanodeUuid();
+ }
+
+ @Metric("Return the Layout Version for the volume")
+ public int getLayoutVersion() {
+ return volume.getLayoutVersion();
+ }
+
+ @Metric("Returns the Volume Type")
+ public String getVolumeType() {
+ return volume.getType().name();
+ }
+
+ public String getMetricsSourceName() {
+ return metricsSourceName;
+ }
+
+ /**
+ * Test conservative avail space.
+ * |----used----| (avail) |++++++++reserved++++++++|
+ * |<------- capacity ------->|
+ * |<------------------- Total capacity -------------->|
+ * A) avail = capacity - used
+ * B) capacity = used + avail
+ * C) Total capacity = used + avail + reserved
+ */
+
+ /**
+ * Return the Storage type for the Volume.
+ */
+ @Metric("Returns the Used space")
+ public long getUsed() {
+ spaceUsed = volume.getVolumeInfo().getScmUsed();
+ return spaceUsed;
Review Comment:
```suggestion
return volume.getVolumeInfo().getScmUsed();
```
The variable isn't needed. You can return the value directly.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume Info stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics",
+ context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String volumeRootStr;
+ private HddsVolume volume;
+
+ private long spaceUsed;
+ private long spaceAvailable;
+ private long spaceReserved;
+
+
+ /**
+ * @param identifier Typically, path to volume root. e.g. /data/hdds
+ */
+ public VolumeInfoStats(String identifier, HddsVolume ref) {
+ this.metricsSourceName +=
+ '-' + identifier + StorageVolume.VolumeType.DATA_VOLUME.toString();
+ this.volumeRootStr = identifier;
+ this.volume = ref;
+ init();
+ }
+
+ public void init() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.register(metricsSourceName, "Volume Info Statistics", this);
+ }
+
+ public void unregister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(metricsSourceName);
+ }
+
+ @Metric("Metric to return the Storage Type")
+ public String getStorageType() {
+ return volume.getStorageType().toString();
+ }
+
+ @Metric("Returns the Directory name for the volume")
+ public String getStorageDirectory() {
+ return volume.getStorageDir().toString();
+ }
+
+ @Metric("Return the DataNode UID for the respective volume")
+ public String getDatanodeUuid() {
+ return volume.getDatanodeUuid();
+ }
+
+ @Metric("Return the Layout Version for the volume")
+ public int getLayoutVersion() {
+ return volume.getLayoutVersion();
+ }
+
+ @Metric("Returns the Volume Type")
+ public String getVolumeType() {
+ return volume.getType().name();
+ }
+
+ public String getMetricsSourceName() {
+ return metricsSourceName;
+ }
+
+ /**
+ * Test conservative avail space.
+ * |----used----| (avail) |++++++++reserved++++++++|
+ * |<------- capacity ------->|
+ * |<------------------- Total capacity -------------->|
+ * A) avail = capacity - used
+ * B) capacity = used + avail
+ * C) Total capacity = used + avail + reserved
+ */
+
+ /**
+ * Return the Storage type for the Volume.
+ */
+ @Metric("Returns the Used space")
+ public long getUsed() {
+ spaceUsed = volume.getVolumeInfo().getScmUsed();
+ return spaceUsed;
+ }
+
+ /**
+ * Return the Total Available capacity of the Volume.
+ */
+ @Metric("Returns the Available space")
+ public long getAvailable() {
+ spaceAvailable = volume.getVolumeInfo().getAvailable();
+ return spaceAvailable;
+ }
+
+ /**
+ * Return the Total Reserved of the Volume.
+ */
+ @Metric("Fetches the Reserved Space")
+ public long getReserved() {
+ spaceReserved = volume.getVolumeInfo().getReservedInBytes();
+ return spaceReserved;
+ }
+
+ /**
+ * Return the Total capacity of the Volume.
+ */
+ @Metric("Returns the Capacity of the Volume")
+ public long getCapacity() {
+ return spaceUsed + spaceAvailable;
+ }
+
+ /**
+ * Return the Total capacity of the Volume.
+ */
+ @Metric("Returns the Total Capacity of the Volume")
+ public long getTotalCapacity() {
+ return (spaceUsed + spaceAvailable + spaceReserved);
Review Comment:
```suggestion
return getUsed() + getAvailable() + getReserved();
```
You should return from the three helper methods instead. Otherwise the variables won't be updated.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume Info stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics",
+ context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String volumeRootStr;
+ private HddsVolume volume;
+
+ private long spaceUsed;
+ private long spaceAvailable;
+ private long spaceReserved;
Review Comment:
```suggestion
```
I believe these three variable are not needed.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume Info stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics",
+ context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String volumeRootStr;
+ private HddsVolume volume;
+
+ private long spaceUsed;
+ private long spaceAvailable;
+ private long spaceReserved;
+
+
+ /**
+ * @param identifier Typically, path to volume root. e.g. /data/hdds
+ */
+ public VolumeInfoStats(String identifier, HddsVolume ref) {
+ this.metricsSourceName +=
+ '-' + identifier + StorageVolume.VolumeType.DATA_VOLUME.toString();
+ this.volumeRootStr = identifier;
+ this.volume = ref;
+ init();
+ }
+
+ public void init() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.register(metricsSourceName, "Volume Info Statistics", this);
+ }
+
+ public void unregister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(metricsSourceName);
+ }
+
+ @Metric("Metric to return the Storage Type")
+ public String getStorageType() {
+ return volume.getStorageType().toString();
+ }
+
+ @Metric("Returns the Directory name for the volume")
+ public String getStorageDirectory() {
+ return volume.getStorageDir().toString();
+ }
+
+ @Metric("Return the DataNode UID for the respective volume")
+ public String getDatanodeUuid() {
+ return volume.getDatanodeUuid();
+ }
+
+ @Metric("Return the Layout Version for the volume")
+ public int getLayoutVersion() {
+ return volume.getLayoutVersion();
+ }
+
+ @Metric("Returns the Volume Type")
+ public String getVolumeType() {
+ return volume.getType().name();
+ }
+
+ public String getMetricsSourceName() {
+ return metricsSourceName;
+ }
+
+ /**
+ * Test conservative avail space.
+ * |----used----| (avail) |++++++++reserved++++++++|
+ * |<------- capacity ------->|
+ * |<------------------- Total capacity -------------->|
+ * A) avail = capacity - used
+ * B) capacity = used + avail
+ * C) Total capacity = used + avail + reserved
+ */
+
+ /**
+ * Return the Storage type for the Volume.
+ */
+ @Metric("Returns the Used space")
+ public long getUsed() {
+ spaceUsed = volume.getVolumeInfo().getScmUsed();
+ return spaceUsed;
+ }
+
+ /**
+ * Return the Total Available capacity of the Volume.
+ */
+ @Metric("Returns the Available space")
+ public long getAvailable() {
+ spaceAvailable = volume.getVolumeInfo().getAvailable();
+ return spaceAvailable;
+ }
+
+ /**
+ * Return the Total Reserved of the Volume.
+ */
+ @Metric("Fetches the Reserved Space")
+ public long getReserved() {
+ spaceReserved = volume.getVolumeInfo().getReservedInBytes();
+ return spaceReserved;
+ }
+
+ /**
+ * Return the Total capacity of the Volume.
+ */
+ @Metric("Returns the Capacity of the Volume")
+ public long getCapacity() {
+ return spaceUsed + spaceAvailable;
Review Comment:
```suggestion
return getUsed() + getAvailable();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r883075875
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume Info stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics",
+ context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String volumeRootStr;
+ private HddsVolume volume;
+
+ private long spaceUsed;
+ private long spaceAvailable;
+ private long spaceReserved;
Review Comment:
That's a good catch! will do!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r887270903
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java:
##########
@@ -332,19 +352,22 @@ public void shutdown() {
if (volumeIOStats != null) {
volumeIOStats.unregister();
}
+ if (volumeInfoMetrics != null) {
+ volumeInfoMetrics.unregister();
+ }
}
/**
* VolumeState represents the different states a HddsVolume can be in.
* NORMAL => Volume can be used for storage
* FAILED => Volume has failed due and can no longer be used for
- * storing containers.
Review Comment:
Yes I am sorry about that, Ill revert back to the previous styling and update the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jojochuang commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
jojochuang commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r887251507
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java:
##########
@@ -332,19 +352,22 @@ public void shutdown() {
if (volumeIOStats != null) {
volumeIOStats.unregister();
}
+ if (volumeInfoMetrics != null) {
+ volumeInfoMetrics.unregister();
+ }
}
/**
* VolumeState represents the different states a HddsVolume can be in.
* NORMAL => Volume can be used for storage
* FAILED => Volume has failed due and can no longer be used for
- * storing containers.
Review Comment:
This is fine but please refrain from updating code style in unrelated code next time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r878007670
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableGaugeLong;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume IO stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics", context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String VolumeRootStr;
+ private HddsVolume volume;
+
+ private @Metric MutableGaugeLong spaceUsed;
+ private @Metric MutableGaugeLong spaceAvailable;
+ private @Metric MutableGaugeLong spaceReserved;
+ private @Metric MutableGaugeLong capacity;
+ private @Metric MutableGaugeLong totalCapacity;
+
+
+ @Deprecated
+ public VolumeInfoStats() {
+ init();
+ }
+
+ /**
+ * @param identifier Typically, path to volume root. e.g. /data/hdds
+ */
+ public VolumeInfoStats(String identifier, HddsVolume ref) {
+ this.metricsSourceName += '-' + identifier;
+ this.VolumeRootStr = identifier;
+ this.volume = ref;
+ init();
+ }
+
+ public void init() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.register(metricsSourceName, "Volume Info Statistics", this);
+ spaceUsed.set(volume.getVolumeInfo().getScmUsed());
+ spaceAvailable.set(volume.getVolumeInfo().getAvailable());
+ spaceReserved.set(volume.getVolumeInfo().getReservedInBytes());
+ capacity.set(spaceUsed.value() + spaceAvailable.value());
+ totalCapacity.set(
+ spaceUsed.value() + spaceAvailable.value() + spaceReserved.value());
+ }
+
+ public void unregister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(metricsSourceName);
+ }
+
+ public String getMetricsSourceName() {
+ return metricsSourceName;
+ }
+
+ /**
+ * Test conservative avail space.
+ * |----used----| (avail) |++++++++reserved++++++++|
+ * |<------- capacity ------->|
+ * |<------------------- Total capacity -------------->|
+ * A) avail = capacity - used
+ * B) avail = fsAvail - Max(reserved - other, 0);
+ */
Review Comment:
I have made changes to the comment!
Thanks! for the suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on PR #3430:
URL: https://github.com/apache/ozone/pull/3430#issuecomment-1138184782
@adoroszlai @JyotinderSingh @siddhantsangwan @jojochuang @aryangupta1998
Can you please help review this !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r880080765
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableGaugeLong;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume IO stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics", context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String VolumeRootStr;
+ private HddsVolume volume;
+
+ private @Metric MutableGaugeLong spaceUsed;
+ private @Metric MutableGaugeLong spaceAvailable;
+ private @Metric MutableGaugeLong spaceReserved;
+ private @Metric MutableGaugeLong capacity;
+ private @Metric MutableGaugeLong totalCapacity;
+
+
+ @Deprecated
+ public VolumeInfoStats() {
+ init();
+ }
+
+ /**
+ * @param identifier Typically, path to volume root. e.g. /data/hdds
+ */
+ public VolumeInfoStats(String identifier, HddsVolume ref) {
+ this.metricsSourceName += '-' + identifier;
+ this.VolumeRootStr = identifier;
+ this.volume = ref;
+ init();
+ }
+
+ public void init() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.register(metricsSourceName, "Volume Info Statistics", this);
+ spaceUsed.set(volume.getVolumeInfo().getScmUsed());
+ spaceAvailable.set(volume.getVolumeInfo().getAvailable());
+ spaceReserved.set(volume.getVolumeInfo().getReservedInBytes());
+ capacity.set(spaceUsed.value() + spaceAvailable.value());
+ totalCapacity.set(
+ spaceUsed.value() + spaceAvailable.value() + spaceReserved.value());
Review Comment:
I have removed them
Please have a look now !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #3430: HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #3430:
URL: https://github.com/apache/ozone/pull/3430#discussion_r884085978
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoStats.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * This class is used to track Volume Info stats for each HDDS Volume.
+ */
+@Metrics(about = "Ozone Volume Information Metrics",
+ context = OzoneConsts.OZONE)
+public class VolumeInfoStats {
+
+ private String metricsSourceName = VolumeInfoStats.class.getSimpleName();
+ private String volumeRootStr;
+ private HddsVolume volume;
+
+ private long spaceUsed;
+ private long spaceAvailable;
+ private long spaceReserved;
+
+
+ /**
+ * @param identifier Typically, path to volume root. e.g. /data/hdds
+ */
+ public VolumeInfoStats(String identifier, HddsVolume ref) {
+ this.metricsSourceName +=
+ '-' + identifier + StorageVolume.VolumeType.DATA_VOLUME.toString();
+ this.volumeRootStr = identifier;
+ this.volume = ref;
+ init();
+ }
+
+ public void init() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.register(metricsSourceName, "Volume Info Statistics", this);
+ }
+
+ public void unregister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(metricsSourceName);
+ }
+
+ @Metric("Metric to return the Storage Type")
+ public String getStorageType() {
+ return volume.getStorageType().toString();
+ }
+
+ @Metric("Returns the Directory name for the volume")
+ public String getStorageDirectory() {
+ return volume.getStorageDir().toString();
+ }
+
+ @Metric("Return the DataNode UID for the respective volume")
+ public String getDatanodeUuid() {
+ return volume.getDatanodeUuid();
+ }
+
+ @Metric("Return the Layout Version for the volume")
+ public int getLayoutVersion() {
+ return volume.getLayoutVersion();
+ }
+
+ @Metric("Returns the Volume Type")
+ public String getVolumeType() {
+ return volume.getType().name();
+ }
+
+ public String getMetricsSourceName() {
+ return metricsSourceName;
+ }
+
+ /**
+ * Test conservative avail space.
+ * |----used----| (avail) |++++++++reserved++++++++|
+ * |<------- capacity ------->|
+ * |<------------------- Total capacity -------------->|
+ * A) avail = capacity - used
+ * B) capacity = used + avail
+ * C) Total capacity = used + avail + reserved
+ */
+
+ /**
+ * Return the Storage type for the Volume.
+ */
+ @Metric("Returns the Used space")
+ public long getUsed() {
+ spaceUsed = volume.getVolumeInfo().getScmUsed();
+ return spaceUsed;
+ }
+
+ /**
+ * Return the Total Available capacity of the Volume.
+ */
+ @Metric("Returns the Available space")
+ public long getAvailable() {
+ spaceAvailable = volume.getVolumeInfo().getAvailable();
+ return spaceAvailable;
+ }
+
+ /**
+ * Return the Total Reserved of the Volume.
+ */
+ @Metric("Fetches the Reserved Space")
+ public long getReserved() {
+ spaceReserved = volume.getVolumeInfo().getReservedInBytes();
+ return spaceReserved;
+ }
+
+ /**
+ * Return the Total capacity of the Volume.
+ */
+ @Metric("Returns the Capacity of the Volume")
+ public long getCapacity() {
+ return spaceUsed + spaceAvailable;
Review Comment:
Updated it !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org