You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "devmadhuu (via GitHub)" <gi...@apache.org> on 2023/06/05 09:16:53 UTC
[GitHub] [ozone] devmadhuu commented on a diff in pull request #4764: HDDS-8585. Recon - API for Number of open keys and amount of data mapped to such keys.
devmadhuu commented on code in PR #4764:
URL: https://github.com/apache/ozone/pull/4764#discussion_r1217760214
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java:
##########
@@ -211,10 +223,45 @@ public Response getOpenKeyInfo(
break;
}
}
+ // Populate the clusterSummary map
+ createClusterSummary(clusterSummary);
+
+ openKeyInsightInfo.setClusterSummary(clusterSummary);
+
openKeyInsightInfo.setLastKey(lastKey);
return Response.ok(openKeyInsightInfo).build();
}
+ private void createClusterSummary(Map<String, Object> clusterSummary) {
+ try {
+ long replicatedSizeOpenKey = globalStatsDao.findById(OmTableInsightTask
+ .getReplicatedSizeKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long replicatedSizeOpenFile = globalStatsDao.findById(OmTableInsightTask
+ .getReplicatedSizeKeyFromTable(OPEN_FILE_TABLE)).getValue();
+ long unreplicatedSizeOpenKey = globalStatsDao.findById(OmTableInsightTask
+ .getUnReplicatedSizeKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long unreplicatedSizeOpenFile = globalStatsDao.findById(OmTableInsightTask
+ .getUnReplicatedSizeKeyFromTable(OPEN_FILE_TABLE)).getValue();
+ long openKeyCountForKeyTable = globalStatsDao.findById(OmTableInsightTask
+ .getTableCountKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long openKeyCountForFileTable = globalStatsDao.findById(OmTableInsightTask
+ .getTableCountKeyFromTable(OPEN_FILE_TABLE)).getValue();
+
+ // Calculate the total number of open keys
+ clusterSummary.put("totalOpenKeys",
+ openKeyCountForKeyTable + openKeyCountForFileTable);
+ // Calculate the total replicated and unreplicated sizes
+ clusterSummary.put("totalReplicatedDataSize",
+ replicatedSizeOpenKey + replicatedSizeOpenFile);
+ clusterSummary.put("totalUnreplicatedDataSize",
+ unreplicatedSizeOpenKey + unreplicatedSizeOpenFile);
+ } catch (NullPointerException e) {
+ // Handle the null pointer exception
Review Comment:
What kind of NPE handling we are doing here ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java:
##########
@@ -211,10 +223,45 @@ public Response getOpenKeyInfo(
break;
}
}
+ // Populate the clusterSummary map
+ createClusterSummary(clusterSummary);
+
+ openKeyInsightInfo.setClusterSummary(clusterSummary);
+
openKeyInsightInfo.setLastKey(lastKey);
return Response.ok(openKeyInsightInfo).build();
}
+ private void createClusterSummary(Map<String, Object> clusterSummary) {
+ try {
+ long replicatedSizeOpenKey = globalStatsDao.findById(OmTableInsightTask
+ .getReplicatedSizeKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long replicatedSizeOpenFile = globalStatsDao.findById(OmTableInsightTask
+ .getReplicatedSizeKeyFromTable(OPEN_FILE_TABLE)).getValue();
+ long unreplicatedSizeOpenKey = globalStatsDao.findById(OmTableInsightTask
+ .getUnReplicatedSizeKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long unreplicatedSizeOpenFile = globalStatsDao.findById(OmTableInsightTask
+ .getUnReplicatedSizeKeyFromTable(OPEN_FILE_TABLE)).getValue();
+ long openKeyCountForKeyTable = globalStatsDao.findById(OmTableInsightTask
+ .getTableCountKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long openKeyCountForFileTable = globalStatsDao.findById(OmTableInsightTask
+ .getTableCountKeyFromTable(OPEN_FILE_TABLE)).getValue();
+
+ // Calculate the total number of open keys
+ clusterSummary.put("totalOpenKeys",
+ openKeyCountForKeyTable + openKeyCountForFileTable);
+ // Calculate the total replicated and unreplicated sizes
+ clusterSummary.put("totalReplicatedDataSize",
Review Comment:
Based on these naming of attributes, please change the "`replicatedTotal`" and "`unreplicatedTotal`" attributes names as well to keep consistency in "`KeyInsightInfoResponse`" class. We can change to "`replicatedDataSize`" and "`unreplicatedDataSize`"
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java:
##########
@@ -211,10 +223,45 @@ public Response getOpenKeyInfo(
break;
}
}
+ // Populate the clusterSummary map
+ createClusterSummary(clusterSummary);
+
+ openKeyInsightInfo.setClusterSummary(clusterSummary);
+
openKeyInsightInfo.setLastKey(lastKey);
return Response.ok(openKeyInsightInfo).build();
}
+ private void createClusterSummary(Map<String, Object> clusterSummary) {
+ try {
+ long replicatedSizeOpenKey = globalStatsDao.findById(OmTableInsightTask
+ .getReplicatedSizeKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long replicatedSizeOpenFile = globalStatsDao.findById(OmTableInsightTask
+ .getReplicatedSizeKeyFromTable(OPEN_FILE_TABLE)).getValue();
+ long unreplicatedSizeOpenKey = globalStatsDao.findById(OmTableInsightTask
+ .getUnReplicatedSizeKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long unreplicatedSizeOpenFile = globalStatsDao.findById(OmTableInsightTask
+ .getUnReplicatedSizeKeyFromTable(OPEN_FILE_TABLE)).getValue();
+ long openKeyCountForKeyTable = globalStatsDao.findById(OmTableInsightTask
+ .getTableCountKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long openKeyCountForFileTable = globalStatsDao.findById(OmTableInsightTask
+ .getTableCountKeyFromTable(OPEN_FILE_TABLE)).getValue();
+
+ // Calculate the total number of open keys
+ clusterSummary.put("totalOpenKeys",
+ openKeyCountForKeyTable + openKeyCountForFileTable);
+ // Calculate the total replicated and unreplicated sizes
+ clusterSummary.put("totalReplicatedDataSize",
+ replicatedSizeOpenKey + replicatedSizeOpenFile);
+ clusterSummary.put("totalUnreplicatedDataSize",
+ unreplicatedSizeOpenKey + unreplicatedSizeOpenFile);
+ } catch (NullPointerException e) {
+ // Handle the null pointer exception
+ e.printStackTrace();
Review Comment:
Its better to avoid printing full stack trace. Use Logger and print useful info.
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestOmTableInsightTask.java:
##########
@@ -155,6 +193,56 @@ public void testProcess() {
assertEquals(5L, getCountForTable(DELETED_TABLE));
}
+ @Test
+ public void testProcessForSize() {
Review Comment:
Write and update test cases in "org.apache.hadoop.ozone.recon.api.TestOmDBInsightEndPoint" class
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java:
##########
@@ -211,10 +223,45 @@ public Response getOpenKeyInfo(
break;
}
}
+ // Populate the clusterSummary map
+ createClusterSummary(clusterSummary);
+
+ openKeyInsightInfo.setClusterSummary(clusterSummary);
+
openKeyInsightInfo.setLastKey(lastKey);
return Response.ok(openKeyInsightInfo).build();
}
+ private void createClusterSummary(Map<String, Object> clusterSummary) {
+ try {
+ long replicatedSizeOpenKey = globalStatsDao.findById(OmTableInsightTask
+ .getReplicatedSizeKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long replicatedSizeOpenFile = globalStatsDao.findById(OmTableInsightTask
+ .getReplicatedSizeKeyFromTable(OPEN_FILE_TABLE)).getValue();
+ long unreplicatedSizeOpenKey = globalStatsDao.findById(OmTableInsightTask
+ .getUnReplicatedSizeKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long unreplicatedSizeOpenFile = globalStatsDao.findById(OmTableInsightTask
+ .getUnReplicatedSizeKeyFromTable(OPEN_FILE_TABLE)).getValue();
+ long openKeyCountForKeyTable = globalStatsDao.findById(OmTableInsightTask
+ .getTableCountKeyFromTable(OPEN_KEY_TABLE)).getValue();
+ long openKeyCountForFileTable = globalStatsDao.findById(OmTableInsightTask
+ .getTableCountKeyFromTable(OPEN_FILE_TABLE)).getValue();
+
+ // Calculate the total number of open keys
+ clusterSummary.put("totalOpenKeys",
Review Comment:
Please update the javadoc as well of API "api/v1/keys/open"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org