You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "tanvipenumudy (via GitHub)" <gi...@apache.org> on 2023/09/19 13:25:24 UTC
[GitHub] [ozone] tanvipenumudy opened a new pull request, #5322: HDDS-9272. Performance improvement for OM's sort datanodes
tanvipenumudy opened a new pull request, #5322:
URL: https://github.com/apache/ozone/pull/5322
## What changes were proposed in this pull request?
After the implementation of [HDDS-8300](https://issues.apache.org/jira/browse/HDDS-8300), get key metadata API in OM sorts datanodes to allow clients to optimize data I/O based on locality. However, as of today, this requires OM to make additional calls to SCM for sorting the datanodes.
The get key metadata API makes 2 separate calls to SCM: one to retrieve the container pipelines and the second responsible for sorting the datanodes in the container pipelines, as illustrated in the code snippet [KeyManagerImpl#getKeyInfo](https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L1946-L1959):
```
if (!args.isHeadOp()) {
...
// get container pipeline info from cache.
captureLatencyNs(metrics.getGetKeyInfoRefreshLocationLatencyNs(),
() -> refreshPipelineFromCache(value,
args.isForceUpdateContainerCacheFromSCM()));
if (args.getSortDatanodes()) {
sortDatanodes(clientAddress, value);
}
```
For performance enhancement, these two calls can be combined into one allowing the result to be cached in the existing `containerLocationCache` in `ScmClient`. More details on the analysis and implementation can be found [here](https://issues.apache.org/jira/browse/HDDS-9272).
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9272
## How was this patch tested?
The existing tests should cover the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] tanvipenumudy commented on pull request #5322: HDDS-9272. Performance improvement for OM's sort datanodes
Posted by "tanvipenumudy (via GitHub)" <gi...@apache.org>.
tanvipenumudy commented on PR #5322:
URL: https://github.com/apache/ozone/pull/5322#issuecomment-1736790421
Closing this PR as we are working on an alternate approach: [HDDS-9343](https://issues.apache.org/jira/browse/HDDS-9343). Shift sortDatanodes logic to OM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] tanvipenumudy closed pull request #5322: HDDS-9272. Performance improvement for OM's sort datanodes
Posted by "tanvipenumudy (via GitHub)" <gi...@apache.org>.
tanvipenumudy closed pull request #5322: HDDS-9272. Performance improvement for OM's sort datanodes
URL: https://github.com/apache/ozone/pull/5322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #5322: HDDS-9272. Performance improvement for OM's sort datanodes
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5322:
URL: https://github.com/apache/ozone/pull/5322#issuecomment-1736205146
We can close this PR while we work on the alternative proposal discuss in the jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] tanvipenumudy commented on pull request #5322: HDDS-9272. Performance improvement for OM's sort datanodes
Posted by "tanvipenumudy (via GitHub)" <gi...@apache.org>.
tanvipenumudy commented on PR #5322:
URL: https://github.com/apache/ozone/pull/5322#issuecomment-1726026777
Taking a look at the Git CI/CD failing checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #5322: HDDS-9272. Performance improvement for OM's sort datanodes
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5322:
URL: https://github.com/apache/ozone/pull/5322#issuecomment-1730034201
Moving the discussion to the jira for now, till the design issues are resolved and we can jump to the code review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org