You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Uma Maheswara Rao G <ha...@gmail.com> on 2018/06/27 22:21:52 UTC
Re: [DISCUSS] Merge Storage Policy Satisfier (SPS) [HDFS-10285]
feature branch to trunk
Hi All,
After long discussions(offline and on JIRA) on SPS, we came to a
conclusion on JIRA(HDFS-10285) that, we will go ahead with External SPS
merge in first phase. In this phase process will not be running inside
Namenode.
We will continue discussion on Internal SPS. Current code base supports
both internal and external option. We have review comments for Internal
which needs some additional works for analysis and testing etc. We will
move Internal SPS work to under HDFS-12226 (Follow-on work for SPS in NN)
We are working on cleanup task HDFS-13076 for the merge. .
For more clarity on Internal and External SPS proposal thoughts, please
refer to JIRA HDFS-10285.
If there are no objections with this, I will go ahead for voting soon.
Regards,
Uma
On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G <ha...@gmail.com>
wrote:
> Update: We worked on the review comments and additional JIRAs above
> mentioned.
>
> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
> planned to take up the support for recursive API support. HDFS-12291<
> https://issues.apache.org/jira/browse/HDFS-12291>
>
> We provided the recursive API support now.
>
> >2. Xattr optimizations HDFS-12225<https://issues.apac
> he.org/jira/browse/HDFS-12225>
> Improved this portion as well
>
> >3. Few other review comments already fixed and committed HDFS-12214<
> https://issues.apache.org/jira/browse/HDFS-12214>
> Fixed the comments.
>
> We are continuing to test the feature and working so far well. Also we
> uploaded a combined patch and got the good QA report.
>
> If there are no further objections, we would like to go for merge vote
> tomorrow. Please by default this feature will be disabled.
>
> Regards,
> Uma
>
> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com> wrote:
>
>> Hi Andrew,
>>
>> >Great to hear. It'd be nice to define which use cases are met by the
>> current version of SPS, and which will be handled after the merge.
>> After the discussions in JIRA, we planned to support recursive API as
>> well. The primary use cases we planned was for Hbase. Please check next
>> point for use case details.
>>
>> >A bit more detail in the design doc on how HBase would use this feature
>> would also be helpful. Is there an HBase JIRA already?
>> Please find the usecase details at this comment in JIRA:
>> https://issues.apache.org/jira/browse/HDFS-10285?focusedComm
>> entId=16120227&page=com.atlassian.jira.plugin.system.issueta
>> bpanels:comment-tabpanel#comment-16120227
>>
>> >I also spent some more time with the design doc and posted a few
>> questions on the JIRA.
>> Thank you for the reviews.
>>
>> To summarize the discussions in JIRA:
>> 1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>> planned to take up the support for recursive API support. HDFS-12291<
>> https://issues.apache.org/jira/browse/HDFS-12291> (Rakesh started the
>> work on it)
>> 2. Xattr optimizations HDFS-12225<https://issues.apac
>> he.org/jira/browse/HDFS-12225> (Patch available)
>> 3. Few other review comments already fixed and committed HDFS-12214<
>> https://issues.apache.org/jira/browse/HDFS-12214>
>>
>> For tracking the follow-up tasks we filed JIRA HDFS-12226, they should
>> not be critical for merge.
>>
>> Regards,
>> Uma
>>
>> From: Andrew Wang <andrew.wang@cloudera.com<mailto:
>> andrew.wang@cloudera.com>>
>> Date: Friday, July 28, 2017 at 11:33 AM
>> To: Uma Gangumalla <uma.gangumalla@intel.com<mailto:
>> uma.gangumalla@intel.com>>
>> Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
>> hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Subject: Re: [DISCUSS] Merge Storage Policy Satisfier (SPS) [HDFS-10285]
>> feature branch to trunk
>>
>> Hi Uma,
>>
>> > If there are still plans to make changes that affect compatibility (the
>> hybrid RPC and bulk DN work mentioned sound like they would), then we can
>> cut branch-3 first, or wait to merge until after these tasks are finished.
>> [Uma] We don’t see that 2 items as high priority for the feature. Users
>> would be able to use the feature with current code base and API. So, we
>> would consider them after branch-3 only. That should be perfectly fine IMO.
>> The current API is very much useful for Hbase scenario. In Hbase case, they
>> will rename files under to different policy directory. They will not set
>> the policies always. So, when rename files under to different policy
>> directory, they can simply call satisfyStoragePolicy, they don’t need any
>> hybrid API.
>>
>> Great to hear. It'd be nice to define which usecases are met by the
>> current version of SPS, and which will be handled after the merge.
>>
>> A bit more detail in the design doc on how HBase would use this feature
>> would also be helpful. Is there an HBase JIRA already?
>>
>> I also spent some more time with the design doc and posted a few
>> questions on the JIRA.
>>
>> Best,
>> Andrew
>>
>
>
答复: [DISCUSS] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk
Posted by "Lin,Yiqun(vip.com)" <yi...@vipshop.com>.
Hi Uma,
Seeing the discussion under JIRA HDFS-10285, external SPS running will be a recommend way for the users, right? As you also mentioned, there are some additional works need to test and be integrated. One more question from me, that maybe also cared by other developers, what's the major differences between internal SPS and external SPS? Just not to affect NN running?
Could you describe a little more for this?
Thanks
Yiqun
-----邮件原件-----
发件人: Uma Maheswara Rao G [mailto:hadoop.uma@gmail.com]
发送时间: 2018年6月28日 6:22
收件人: hdfs-dev@hadoop.apache.org
主题: Re: [DISCUSS] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk
Hi All,
After long discussions(offline and on JIRA) on SPS, we came to a conclusion on JIRA(HDFS-10285) that, we will go ahead with External SPS merge in first phase. In this phase process will not be running inside Namenode.
We will continue discussion on Internal SPS. Current code base supports both internal and external option. We have review comments for Internal which needs some additional works for analysis and testing etc. We will move Internal SPS work to under HDFS-12226 (Follow-on work for SPS in NN) We are working on cleanup task HDFS-13076 for the merge. .
For more clarity on Internal and External SPS proposal thoughts, please refer to JIRA HDFS-10285.
If there are no objections with this, I will go ahead for voting soon.
Regards,
Uma
On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G <ha...@gmail.com>
wrote:
> Update: We worked on the review comments and additional JIRAs above
> mentioned.
>
> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
> planned to take up the support for recursive API support. HDFS-12291<
> https://issues.apache.org/jira/browse/HDFS-12291>
>
> We provided the recursive API support now.
>
> >2. Xattr optimizations HDFS-12225<https://issues.apac
> he.org/jira/browse/HDFS-12225>
> Improved this portion as well
>
> >3. Few other review comments already fixed and committed HDFS-12214<
> https://issues.apache.org/jira/browse/HDFS-12214>
> Fixed the comments.
>
> We are continuing to test the feature and working so far well. Also we
> uploaded a combined patch and got the good QA report.
>
> If there are no further objections, we would like to go for merge vote
> tomorrow. Please by default this feature will be disabled.
>
> Regards,
> Uma
>
> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com> wrote:
>
>> Hi Andrew,
>>
>> >Great to hear. It'd be nice to define which use cases are met by the
>> current version of SPS, and which will be handled after the merge.
>> After the discussions in JIRA, we planned to support recursive API as
>> well. The primary use cases we planned was for Hbase. Please check
>> next point for use case details.
>>
>> >A bit more detail in the design doc on how HBase would use this
>> >feature
>> would also be helpful. Is there an HBase JIRA already?
>> Please find the usecase details at this comment in JIRA:
>> https://issues.apache.org/jira/browse/HDFS-10285?focusedComm
>> entId=16120227&page=com.atlassian.jira.plugin.system.issueta
>> bpanels:comment-tabpanel#comment-16120227
>>
>> >I also spent some more time with the design doc and posted a few
>> questions on the JIRA.
>> Thank you for the reviews.
>>
>> To summarize the discussions in JIRA:
>> 1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>> planned to take up the support for recursive API support. HDFS-12291<
>> https://issues.apache.org/jira/browse/HDFS-12291> (Rakesh started the
>> work on it) 2. Xattr optimizations HDFS-12225<https://issues.apac
>> he.org/jira/browse/HDFS-12225> (Patch available) 3. Few other review
>> comments already fixed and committed HDFS-12214<
>> https://issues.apache.org/jira/browse/HDFS-12214>
>>
>> For tracking the follow-up tasks we filed JIRA HDFS-12226, they
>> should not be critical for merge.
>>
>> Regards,
>> Uma
>>
>> From: Andrew Wang <andrew.wang@cloudera.com<mailto:
>> andrew.wang@cloudera.com>>
>> Date: Friday, July 28, 2017 at 11:33 AM
>> To: Uma Gangumalla <uma.gangumalla@intel.com<mailto:
>> uma.gangumalla@intel.com>>
>> Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
>> hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Subject: Re: [DISCUSS] Merge Storage Policy Satisfier (SPS)
>> [HDFS-10285] feature branch to trunk
>>
>> Hi Uma,
>>
>> > If there are still plans to make changes that affect compatibility
>> > (the
>> hybrid RPC and bulk DN work mentioned sound like they would), then we
>> can cut branch-3 first, or wait to merge until after these tasks are finished.
>> [Uma] We don’t see that 2 items as high priority for the feature.
>> Users would be able to use the feature with current code base and
>> API. So, we would consider them after branch-3 only. That should be perfectly fine IMO.
>> The current API is very much useful for Hbase scenario. In Hbase
>> case, they will rename files under to different policy directory.
>> They will not set the policies always. So, when rename files under to
>> different policy directory, they can simply call
>> satisfyStoragePolicy, they don’t need any hybrid API.
>>
>> Great to hear. It'd be nice to define which usecases are met by the
>> current version of SPS, and which will be handled after the merge.
>>
>> A bit more detail in the design doc on how HBase would use this
>> feature would also be helpful. Is there an HBase JIRA already?
>>
>> I also spent some more time with the design doc and posted a few
>> questions on the JIRA.
>>
>> Best,
>> Andrew
>>
>
>
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy.