You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/12/27 12:43:55 UTC
[GitHub] [ozone] devmadhuu opened a new pull request, #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
devmadhuu opened a new pull request, #4129:
URL: https://github.com/apache/ozone/pull/4129
Understand the age of data ( age of buckets, age of objects, age of volumes)
1. This requirement will be limited by the number of buckets (say top 10) to show with their age information as a given volume can have a large number of buckets. Same holds true for showing a limited number of volumes (say top 10) with their age information if root path is given.
2. Top 10 criteria can be determined based on following factors:
3. Age of Top 10 buckets can be listed based on either count of keys or size of buckets, if path is upto a specific volume.
4. Age of top 10 volumes can be listed based on either count of buckets or size of volumes if path is given as root .
5. If the path given by the user is upto bucket level, then top 10 keys based on size will be given with their age details.
6. If the path given by the user is upto key or leaf level, then only that key age details will be returned along with size.
https://issues.apache.org/jira/browse/HDDS-7650
## How was this patch tested?
This patch is tested using postman by verifying backend responses:
API details:
If Path is Bucket:
Keys
TopN Keys with Age, Size details
By Count - Give user option to select 10, 20, 30 etc
By Size - Give user option to select With or Without Replica
Dirs
TopN Dirs with Age, Size, dirCount, keycount details
By Count - Give user option to select 10, 20, 30 etc
By Size - Give user option to select With or Without Replica
If Path is Volume:
Buckets
TopN Buckets with Age, Size, dirCount, keycount details
By Count - Give user option to select 10, 20, 30 etc
By Size - Give user option to select With or Without Replica
If Path is Directory:
Keys
TopN Keys with Age, Size details
By Count - Give user option to select 10, 20, 30 etc
By Size - Give user option to select With or Without Replica
Dirs
TopN Dirs with Age, Size, dirCount, keycount details
By Count - Give user option to select 10, 20, 30 etc
By Size - Give user option to select With or Without Replica
If Path is Key:
Display: Size, Age details
API End Point :
Request Type: GET
API Path: /api/v1/namespace/entitymetrics?path=<path of volume/bucket/dir/key>
Sample Response:
{
"entityInfo":
{ "type": "DIRECTORY", "numVolume": -1, "numBucket": -1, "numDir": 2, "numKey": 5, "status": "OK" }
,
"childEntityMetrics": [
{ "key": false, "path": "/volume1/bucket1/dir1/dir2", "size": 86060, "sizeWithReplica": 258180, "isKey": false, "keyCount": 4, "bucketCount": -1, "dirCount": 1, "volumeCount": -1, "type": "DIRECTORY", "creationTime": 1671801625290, "modificationTime": 1671801625290 }
,
{ "key": true, "path": "/volume1/bucket1/dir1/LICENSE.txt", "size": 21515, "sizeWithReplica": 64545, "isKey": true, "keyCount": 0, "bucketCount": -1, "dirCount": -1, "volumeCount": -1, "type": "KEY", "creationTime": 1671801625290, "modificationTime": 1671801625290 }
],
"status": "OK"
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #4129:
URL: https://github.com/apache/ozone/pull/4129#discussion_r1060263150
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/NamespaceSummaryResponse.java:
##########
@@ -43,6 +43,16 @@ public class NamespaceSummaryResponse {
@JsonProperty("numKey")
private long numTotalKey;
+ /** Age of entity in
+ * human-readable format. */
+ @JsonProperty("age")
+ private long age;
Review Comment:
@sumitagrawl - Thanks for reviewing. It has been fixed. Pls re-review.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java:
##########
@@ -62,6 +70,129 @@ public NSSummaryEndpoint(ReconNamespaceSummaryManager namespaceSummaryManager,
this.reconSCM = reconSCM;
}
+ @GET
+ @Path("/entitymetrics")
+ public Response getEntityMetrics(@QueryParam("path") String path,
+ @DefaultValue("size")
Review Comment:
@sumitagrawl - Thanks for reviewing. It has been fixed. Pls re-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on PR #4129:
URL: https://github.com/apache/ozone/pull/4129#issuecomment-1384824941
@ChenSammi , @duongkame pls review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #4129:
URL: https://github.com/apache/ozone/pull/4129#issuecomment-1375990020
@duongkame can you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smitajoshi12 commented on pull request #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
Posted by "smitajoshi12 (via GitHub)" <gi...@apache.org>.
smitajoshi12 commented on PR #4129:
URL: https://github.com/apache/ozone/pull/4129#issuecomment-1413378384
Can we get mutiple Parent Entity Records?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
URL: https://github.com/apache/ozone/pull/4129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] fapifta commented on a diff in pull request #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
Posted by "fapifta (via GitHub)" <gi...@apache.org>.
fapifta commented on code in PR #4129:
URL: https://github.com/apache/ozone/pull/4129#discussion_r1084756443
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java:
##########
@@ -48,6 +56,8 @@
@AdminOnly
public class NSSummaryEndpoint {
+ public static final String COUNT = "count";
Review Comment:
Shouldn't we have createTime here? Also shouldn't that be cTime or creationTime?
As these three belongs together, wouldn't it worth it to create an enum for the values we accept, and validate and parse the parameters based on the enum?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java:
##########
@@ -116,21 +294,49 @@ public Response getDiskUsage(@QueryParam("path") String path,
if (path == null || path.length() == 0) {
return Response.status(Response.Status.BAD_REQUEST).build();
}
+ return Response.ok(getDuResponse(path, listFile, withReplica)).build();
+ }
+ private DUResponse getDuResponse(String path, boolean listFile,
+ boolean withReplica) throws IOException {
+ AtomicBoolean isError = new AtomicBoolean(false);
Review Comment:
I believe we can use a simple boolean here, even if the forEach runs in parallel mode, the only change to the value is to set it to true, and we just check after execution is done, so we don't need atomicity on it as it does not matter i one or 3k threads sets it to true at once, but correct me if I am wrong.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java:
##########
@@ -116,21 +294,49 @@ public Response getDiskUsage(@QueryParam("path") String path,
if (path == null || path.length() == 0) {
return Response.status(Response.Status.BAD_REQUEST).build();
}
+ return Response.ok(getDuResponse(path, listFile, withReplica)).build();
Review Comment:
After extracting the method here, but even before that, we can remove the @SuppressWarnings for the /du endpoint method in line 287
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java:
##########
@@ -62,6 +72,172 @@ public NSSummaryEndpoint(ReconNamespaceSummaryManager namespaceSummaryManager,
this.reconSCM = reconSCM;
}
+ @GET
+ @Path("/entityinfo")
+ public Response getEntityMetrics(@QueryParam("path") String path,
Review Comment:
Can we add API documentation for this method if we keep it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #4129:
URL: https://github.com/apache/ozone/pull/4129#discussion_r1060263182
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java:
##########
@@ -62,6 +70,129 @@ public NSSummaryEndpoint(ReconNamespaceSummaryManager namespaceSummaryManager,
this.reconSCM = reconSCM;
}
+ @GET
+ @Path("/entitymetrics")
Review Comment:
@sumitagrawl - Thanks for reviewing. It has been fixed. Pls re-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4129:
URL: https://github.com/apache/ozone/pull/4129#issuecomment-1926006585
Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author.
It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.
It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.
If you need ANY help to finish this PR, please [contact the community](https://github.com/apache/hadoop-ozone#contact) on the mailing list or the slack channel."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on PR #4129:
URL: https://github.com/apache/ozone/pull/4129#issuecomment-1365908314
@sumitagrawl @dombizita @smengcl , pls review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on PR #4129:
URL: https://github.com/apache/ozone/pull/4129#issuecomment-1398058989
> LGTM, please add Unit cases
@sumitagrawl - Added test cases. Pls review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4129:
URL: https://github.com/apache/ozone/pull/4129#issuecomment-1890979952
/pending
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
Posted by GitBox <gi...@apache.org>.
sumitagrawl commented on code in PR #4129:
URL: https://github.com/apache/ozone/pull/4129#discussion_r1059944398
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java:
##########
@@ -62,6 +70,129 @@ public NSSummaryEndpoint(ReconNamespaceSummaryManager namespaceSummaryManager,
this.reconSCM = reconSCM;
}
+ @GET
+ @Path("/entitymetrics")
+ public Response getEntityMetrics(@QueryParam("path") String path,
+ @DefaultValue("size")
Review Comment:
orderby should include "age" also a filter and as default
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java:
##########
@@ -62,6 +70,129 @@ public NSSummaryEndpoint(ReconNamespaceSummaryManager namespaceSummaryManager,
this.reconSCM = reconSCM;
}
+ @GET
+ @Path("/entitymetrics")
Review Comment:
uri name can be entityInfo or other better name, not a metrics as its not a metric data.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/NamespaceSummaryResponse.java:
##########
@@ -43,6 +43,16 @@ public class NamespaceSummaryResponse {
@JsonProperty("numKey")
private long numTotalKey;
+ /** Age of entity in
+ * human-readable format. */
+ @JsonProperty("age")
+ private long age;
Review Comment:
can have same name as createTime, UI can calculate age and display name accordingly
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java:
##########
@@ -62,6 +70,129 @@ public NSSummaryEndpoint(ReconNamespaceSummaryManager namespaceSummaryManager,
this.reconSCM = reconSCM;
}
+ @GET
+ @Path("/entitymetrics")
+ public Response getEntityMetrics(@QueryParam("path") String path,
+ @DefaultValue("size")
+ @QueryParam("orderBy") String orderBy,
+ @DefaultValue("10")
+ @QueryParam("count") int count)
+ throws IOException {
+ if (path == null || path.length() == 0) {
+ return Response.status(Response.Status.BAD_REQUEST).build();
+ }
+ EntityMetricsResponse entityMetricsResponse;
+ if (!isInitializationComplete()) {
+ entityMetricsResponse =
+ new EntityMetricsResponse(EntityType.UNKNOWN);
+ entityMetricsResponse.setStatus(ResponseStatus.INITIALIZING);
+ } else {
+ Map<String, List<DUResponse.DiskUsage>> childMetricsListMap =
+ new HashMap<>();
+ NamespaceSummaryResponse nsSummaryData =
+ getNamespaceSummaryResponse(path);
+ DUResponse duResponse = getDuResponse(path, true, true);
+
+ EntityType entityType = nsSummaryData.getEntityType();
+ entityMetricsResponse = new EntityMetricsResponse(entityType);
+
+ if (COUNT.equalsIgnoreCase(orderBy)) {
+ switch (entityType) {
+ case ROOT:
+ childMetricsListMap.put(EntityType.VOLUME.name(),
+ duResponse.getDuData()
+ .stream().sorted(Comparator.comparingLong(
+ DUResponse.DiskUsage::getVolumeCount).reversed())
+ .limit(count).collect(Collectors.toList()));
+ break;
+ case VOLUME:
+ childMetricsListMap.put(EntityType.BUCKET.name(),
+ duResponse.getDuData()
+ .stream().sorted(Comparator.comparingLong(
+ DUResponse.DiskUsage::getBucketCount).reversed())
+ .limit(count).collect(Collectors.toList()));
+ break;
+ case BUCKET:
+ case DIRECTORY:
+ List<DUResponse.DiskUsage> keys = getKeys(duResponse.getDuData());
+ List<DUResponse.DiskUsage> dirs = getDirs(duResponse.getDuData());
+ childMetricsListMap.put(EntityType.KEY.name(),
+ keys.stream().sorted(Comparator.comparingLong(
+ DUResponse.DiskUsage::getKeyCount).reversed())
+ .limit(count).collect(Collectors.toList()));
+ childMetricsListMap.put(EntityType.DIRECTORY.name(),
+ dirs.stream().sorted(Comparator.comparingLong(
+ DUResponse.DiskUsage::getKeyCount).reversed())
+ .limit(count).collect(Collectors.toList()));
+ break;
+ default:
+ childMetricsListMap.put(EntityType.UNKNOWN.name(),
+ duResponse.getDuData()
+ .stream().sorted(Comparator.comparingLong(
+ DUResponse.DiskUsage::getKeyCount).reversed())
+ .limit(count).collect(Collectors.toList()));
+ break;
+ }
+ } else {
+ switch (entityType) {
+ case ROOT:
+ childMetricsListMap.put(EntityType.VOLUME.name(),
+ duResponse.getDuData()
+ .stream().sorted(Comparator.comparingLong(
+ DUResponse.DiskUsage::getSizeWithReplica).reversed())
+ .limit(count).collect(Collectors.toList()));
+ break;
+ case VOLUME:
+ childMetricsListMap.put(EntityType.BUCKET.name(),
+ duResponse.getDuData()
+ .stream().sorted(Comparator.comparingLong(
+ DUResponse.DiskUsage::getSizeWithReplica).reversed())
+ .limit(count).collect(Collectors.toList()));
+ break;
+ case BUCKET:
+ case DIRECTORY:
+ List<DUResponse.DiskUsage> keys = getKeys(duResponse.getDuData());
+ List<DUResponse.DiskUsage> dirs = getDirs(duResponse.getDuData());
+ childMetricsListMap.put(EntityType.KEY.name(),
+ keys.stream().sorted(Comparator.comparingLong(
+ DUResponse.DiskUsage::getSizeWithReplica).reversed())
+ .limit(count).collect(Collectors.toList()));
+ childMetricsListMap.put(EntityType.DIRECTORY.name(),
+ dirs.stream().sorted(Comparator.comparingLong(
+ DUResponse.DiskUsage::getSizeWithReplica).reversed())
+ .limit(count).collect(Collectors.toList()));
+ break;
+ default:
+ childMetricsListMap.put(EntityType.UNKNOWN.name(),
+ duResponse.getDuData()
+ .stream().sorted(Comparator.comparingLong(
+ DUResponse.DiskUsage::getSizeWithReplica).reversed())
+ .limit(count).collect(Collectors.toList()));
+ break;
+ }
+ }
+ entityMetricsResponse.setNsSummaryResponse(nsSummaryData);
+ entityMetricsResponse.setChildMetricsListMap(childMetricsListMap);
Review Comment:
childMetricsList should be part of NsSummary as it belongs to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #4129: HDDS-7650. Recon - Data Management Metrics - Understand the Age of Data.
Posted by "DaveTeng0 (via GitHub)" <gi...@apache.org>.
DaveTeng0 commented on PR #4129:
URL: https://github.com/apache/ozone/pull/4129#issuecomment-1418573718
Hey @devmadhuu ~ there are some new comments. Please help address, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org