You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Genmao Yu (JIRA)" <ji...@apache.org> on 2017/11/17 09:32:00 UTC
[jira] [Comment Edited] (HADOOP-15038) Abstract MetadataStore in
S3Guard into a common module.
[ https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256714#comment-16256714 ]
Genmao Yu edited comment on HADOOP-15038 at 11/17/17 9:31 AM:
--------------------------------------------------------------
[~fabbri] Sorry for the late response, and thanks for your comments.
bq. I originally intended for this to be possible, just didn't want to pollute public hadoop common until it was needed by another client.
What about move it to a separate module? like {{hadoop-metastore}} inside {{hadoop-tools}}.
bq. It might be a good idea to use HADOOP-14098 as a top-level "umbrella" JIRA for your effort, and create subtasks ...
Absolutely yes. Now I did these works offline and made a POC. As soon as I can confirm the validity of metastore (call it this way first) for performance improving, I will begin to provide this feature into Aliyun OSS fs. After all, I want use metastore to improve performance of metadata operation. There are still many blocks, like {{rename}}. It is not a metadata operation issue, but can be done with metastore.
bq. Can you comment on when you will need this?
Maybe, separating and my work can be done in parallel. Is there any concern? Besides, I can help you to do separating work.
was (Author: unclegen):
[~fabbri] Sorry for the late response, and thanks for your comments.
bq. I originally intended for this to be possible, just didn't want to pollute public hadoop common until it was needed by another client.
What about move it to a separate module? like {{hadoop-metastore}} inside {{hadoop-tools}}.
bq. It might be a good idea to use HADOOP-14098 as a top-level "umbrella" JIRA for your effort, and create subtasks ...
Absolutely yes. Now I did these works offline and made a POC. As soon as I can confirm the validity of metastore (call it this way first) for performance improving. After all, I want use metastore to improve performance of metadata operation. There are still many blocks, like {{rename}}. It is not a metadata operation issue, but can be done with metastore.
bq. Can you comment on when you will need this?
Maybe, separating and my work can be done in parallel. Is there any concern? Besides, I can help you to do separating work.
> Abstract MetadataStore in S3Guard into a common module.
> -------------------------------------------------------
>
> Key: HADOOP-15038
> URL: https://issues.apache.org/jira/browse/HADOOP-15038
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Affects Versions: 3.0.0-beta1
> Reporter: Genmao Yu
>
> Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}} into a common module.
> Based on this work, other filesystem or object store can implement their own metastore for optimization (known issues like consistency problem and metadata operation performance). [~stevel@apache.org] and other guys have done many base and great works in {{S3Guard}}. It is very helpful to start work. I did some perf test in HADOOP-14098, and started related work for Aliyun OSS. Indeed there are still works to do for {{S3Guard}}, like metadata cache inconsistent with S3 and so on. It also will be a problem for other object store. However, we can do these works in parallel.
> [~stevel@apache.org] [~fabbri] [~drankye] Any suggestion is appreciated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org