You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Genmao Yu (JIRA)" <ji...@apache.org> on 2017/11/10 12:09:00 UTC
[jira] [Commented] (HADOOP-14098) AliyunOSS: improve the
performance of object metadata operation
[ https://issues.apache.org/jira/browse/HADOOP-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247399#comment-16247399 ]
Genmao Yu commented on HADOOP-14098:
------------------------------------
h3. AliyunOSS FS + metastore POC
1. metastore: a common module of {{MetadataStore}} interface to manager metadata of OSS object. Other filesystem or object store can manager their metadata based on specific implementation of metastore
2. In this POC, I implemented one common metastore based on HDFS, i.e. we use HDFS to manager metadata of object store. One advantage is the semantic consistency, we can use HDFS backened metastore to manage metadata just like use HDFS. Another advantage is this HDFS based metastore is platform independent, i.e. we can use it to manager any object store metadata.
h4. Goals:
- Test if metastore can accelerate the performance of metadata operation.
h4. Non-Goals:
- This test is not a pressure test of cluster. The test result does not indicate any platform or product performance.
| operation|unit|oss fs + metastore|oss fs|hdfs|
|CreateOp| | | | |
| |operations/sec|0.243|0.252|1.84|
| |successes/sec|0.152|0.161|0.324|
|DeleteOp| | | | |
| |operations/sec|52.479|43.499|328.515|
| |successes/sec|19.438|16.947|162.943|
|ListOp| | | | |
| |directory entries/sec|303.461|51.949|386.441|
| |operations/sec|250.05|37.191|235.405|
| |successes/sec|247.85|37.043|233.898|
|MkdirOp| | | | |
| |operations/sec|227.853|73.738|225.388|
| |successes/sec|227.853|73.738|225.388|
|ReadOp| | | | |
| |operations/sec|2.039|1.606|47.426|
| |successes/sec|0.127|0.132|6.222|
|RenameOp| | | | |
| |operations/sec|2.542|2.986|578.972|
| |successes/sec|0.578|0.507|163.965|
|AppendOp| |No Support|No Support| |
With metastore, we can find：
- {{ListOp}} and {{MkdirOp}} have a similar performance with HDFS.
- Others like {{RenameOp}} {{ReadOp}} {{DeleteOp}} still are much worse than HDFS.
> AliyunOSS: improve the performance of object metadata operation
> ---------------------------------------------------------------
>
> Key: HADOOP-14098
> URL: https://issues.apache.org/jira/browse/HADOOP-14098
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
> Affects Versions: 3.0.0-alpha2
> Reporter: Genmao Yu
> Assignee: Genmao Yu
>
> Open this JIRA to research and address the potential request performance issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org