You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Gangumalla, Uma" <um...@intel.com> on 2014/05/14 14:47:14 UTC
[Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Hello HDFS Devs,
I would like to call for a vote to merge the HDFS Extended Attributes (XAttrs) feature from the HDFS-2006 branch to the trunk.
XAttrs are already widely supported on many operating systems, including Linux, Windows, and Mac OS. This will allow storing attributes for HDFS file/directory.
XAttr consist of a name and a value and exist in one of 4 namespaces: user, trusted, security, and system. An XAttr name is prefixed with one of these namespaces, so for example, "user.myxattr".
Consistent with ongoing awareness of Namenode memory usage, the maximum number and size of XAttrs on a file/directory are limited by a configuration parameter.
The design document contains more details and can be found here: https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
Development of this feature has been tracked in JIRA HDFS-2006: https://issues.apache.org/jira/browse/HDFS-2006
All of the development work for the feature is contained in the "HDFS-2006" branch: https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
As last tasks, we are working to support XAttrs via libhdfs, webhdfs as well as other minor improvements.
We intend to finish those enhancements before the vote completes and otherwise we could move them to top-level JIRAs as they can be tracked independently. User document is also ready for this feature.
Here the doc attached in JIRA: https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
The XAttrs feature is backwards-compatible and enabled by default. A cluster administrator can disable it.
Testing:
We've developed more than 70 new tests which cover the XAttrs get, set and remove APIs through DistributedFileSystem and WebHdfsFileSystem, the new XAttr CLI commands, HA, XAttr persistence in the fsimage and related.
Additional testing plans are documented in: https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
Thanks a lot to the contributors who have helped and participated in the branch development.
Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma Maheswara Rao G.
The design document incorporates feedback from many community members: Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb, Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
Code reviewers on individual patches include Chris Nauroth, Alejandro, Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
Also thanks to Dhruba for bringing up this JIRA and thanks to others who participated for discussions.
This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
Here is my +1 to start with.
Regards,
Uma
(umamahesh@apache.org<ma...@apache.org>)
Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
Great job, guys. +1.
I don't think we need to finish libhdfs support before we merge (unless you
want to).
Colin
On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma
<um...@intel.com>wrote:
> Hello HDFS Devs,
> I would like to call for a vote to merge the HDFS Extended Attributes
> (XAttrs) feature from the HDFS-2006 branch to the trunk.
> XAttrs are already widely supported on many operating systems, including
> Linux, Windows, and Mac OS. This will allow storing attributes for HDFS
> file/directory.
> XAttr consist of a name and a value and exist in one of 4 namespaces:
> user, trusted, security, and system. An XAttr name is prefixed with one of
> these namespaces, so for example, "user.myxattr".
> Consistent with ongoing awareness of Namenode memory usage, the maximum
> number and size of XAttrs on a file/directory are limited by a
> configuration parameter.
> The design document contains more details and can be found here:
> https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
> Development of this feature has been tracked in JIRA HDFS-2006:
> https://issues.apache.org/jira/browse/HDFS-2006
> All of the development work for the feature is contained in the
> "HDFS-2006" branch:
> https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
> As last tasks, we are working to support XAttrs via libhdfs, webhdfs as
> well as other minor improvements.
> We intend to finish those enhancements before the vote completes and
> otherwise we could move them to top-level JIRAs as they can be tracked
> independently. User document is also ready for this feature.
> Here the doc attached in JIRA:
> https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
> The XAttrs feature is backwards-compatible and enabled by default. A
> cluster administrator can disable it.
> Testing:
> We've developed more than 70 new tests which cover the XAttrs get, set
> and remove APIs through DistributedFileSystem and WebHdfsFileSystem, the
> new XAttr CLI commands, HA, XAttr persistence in the fsimage and related.
> Additional testing plans are documented in:
> https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
> Thanks a lot to the contributors who have helped and participated in the
> branch development.
> Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma
> Maheswara Rao G.
> The design document incorporates feedback from many community members:
> Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb,
> Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
> Code reviewers on individual patches include Chris Nauroth, Alejandro,
> Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
>
> Also thanks to Dhruba for bringing up this JIRA and thanks to others who
> participated for discussions.
> This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
>
> Here is my +1 to start with.
> Regards,
> Uma
> (umamahesh@apache.org<ma...@apache.org>)
>
>
>
>
Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Posted by Suresh Srinivas <su...@hortonworks.com>.
> I think you're alluding to the recent WebHDFS+ACLs compat JIRA here,
No. My comments was unrelated.
> but I
> don't think xattrs will hit the same issues. The xattr functionality is
> wholly contained within new RPC calls, and AFAIK didn't modify any existing
> client calls. You have to explicitly call "hadoop fs -getxattr" or similar,
> in which case it'll error if you're talking to an old server.
>
Sounds good, as long new client continues to work with the older server.
> Also note that this is just to merge to trunk, we'll be doing further
> compat testing internally before merging to branch-2.
>
Sounds good.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Posted by Andrew Wang <an...@cloudera.com>.
I think you're alluding to the recent WebHDFS+ACLs compat JIRA here, but I
don't think xattrs will hit the same issues. The xattr functionality is
wholly contained within new RPC calls, and AFAIK didn't modify any existing
client calls. You have to explicitly call "hadoop fs -getxattr" or similar,
in which case it'll error if you're talking to an old server.
Also note that this is just to merge to trunk, we'll be doing further
compat testing internally before merging to branch-2.
On Mon, May 19, 2014 at 9:44 AM, Suresh Srinivas <su...@hortonworks.com>wrote:
> NN layout version bump should be fine. This means during rolling upgrade
> downgrade from 2.5 NN to 2.4 NN will not be possible. Only rollback can be
> supported.
>
> HDFS-2006 adds new RPCs. The question is, does the new client that depends
> on this functionality gracefully fallback when talking to old server? Also
> are there any incompatibilities between NN and DN (my guess is there is
> none, given this is NN feature)?
>
>
>
> On Fri, May 16, 2014 at 5:09 PM, Andrew Wang <andrew.wang@cloudera.com
> >wrote:
>
> > Hi Suresh,
> >
> > HDFS-2006 adds new edit log ops (no modifications to existing ops) and a
> > new XAttr INode Feature needs to be serialized to the fsimage. This
> > required doing a NN layout version bump, so a 2.4 NN will not be able to
> > start with these edit logs or images. Thus it's incompatible.
> >
> > I think you're better equipped to comment on rolling upgrades, but based
> on
> > my understanding, I think the above means we should still be able to
> > rolling upgrade between 2.4 and whatever version eventually includes
> xattr
> > support.
> >
> > RPC-wise, HDFS-2006 only adds new RPCs, so I don't think there are any
> > concerns there.
> >
> > Thanks,
> > Andrew
> >
> >
> > On Fri, May 16, 2014 at 2:56 PM, Suresh Srinivas <suresh@hortonworks.com
> > >wrote:
> >
> > > I have not looked at the development closely. With rolling upgrades
> > > feature support in, are there any incompatible changes with this
> feature?
> > >
> > > Sent from phone
> > >
> > > > On May 16, 2014, at 10:30 AM, Chris Nauroth <
> cnauroth@hortonworks.com>
> > > wrote:
> > > >
> > > > +1 for the merge.
> > > >
> > > > I've participated in ongoing design discussions and code reviews on
> > > > individual patches. Yesterday, I completed a final review pass over
> > the
> > > > code in the feature branch, the design document, the end user
> > > documentation
> > > > and the test plan. It looks ready to me.
> > > >
> > > > Thank you to everyone who made contributions on the feature branch.
> > > >
> > > > Chris Nauroth
> > > > Hortonworks
> > > > http://hortonworks.com/
> > > >
> > > >
> > > >
> > > > On Wed, May 14, 2014 at 9:20 AM, Andrew Wang <
> andrew.wang@cloudera.com
> > > >wrote:
> > > >
> > > >> +1 from me as well, I've participated in review and development of
> > this
> > > >> branch and think it's ready for merge.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >>
> > > >> On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma
> > > >> <um...@intel.com>wrote:
> > > >>
> > > >>> Hello HDFS Devs,
> > > >>> I would like to call for a vote to merge the HDFS Extended
> > Attributes
> > > >>> (XAttrs) feature from the HDFS-2006 branch to the trunk.
> > > >>> XAttrs are already widely supported on many operating systems,
> > > >> including
> > > >>> Linux, Windows, and Mac OS. This will allow storing attributes for
> > HDFS
> > > >>> file/directory.
> > > >>> XAttr consist of a name and a value and exist in one of 4
> > namespaces:
> > > >>> user, trusted, security, and system. An XAttr name is prefixed with
> > one
> > > >> of
> > > >>> these namespaces, so for example, "user.myxattr".
> > > >>> Consistent with ongoing awareness of Namenode memory usage, the
> > > maximum
> > > >>> number and size of XAttrs on a file/directory are limited by a
> > > >>> configuration parameter.
> > > >>> The design document contains more details and can be found here:
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
> > > >>> Development of this feature has been tracked in JIRA HDFS-2006:
> > > >>> https://issues.apache.org/jira/browse/HDFS-2006
> > > >>> All of the development work for the feature is contained in the
> > > >>> "HDFS-2006" branch:
> > > >>> https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
> > > >>> As last tasks, we are working to support XAttrs via libhdfs,
> webhdfs
> > as
> > > >>> well as other minor improvements.
> > > >>> We intend to finish those enhancements before the vote completes
> and
> > > >>> otherwise we could move them to top-level JIRAs as they can be
> > tracked
> > > >>> independently. User document is also ready for this feature.
> > > >>> Here the doc attached in JIRA:
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
> > > >>> The XAttrs feature is backwards-compatible and enabled by default.
> A
> > > >>> cluster administrator can disable it.
> > > >>> Testing:
> > > >>> We've developed more than 70 new tests which cover the XAttrs get,
> > set
> > > >>> and remove APIs through DistributedFileSystem and
> WebHdfsFileSystem,
> > > the
> > > >>> new XAttr CLI commands, HA, XAttr persistence in the fsimage and
> > > related.
> > > >>> Additional testing plans are documented in:
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
> > > >>> Thanks a lot to the contributors who have helped and participated
> in
> > > >> the
> > > >>> branch development.
> > > >>> Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma
> > > >>> Maheswara Rao G.
> > > >>> The design document incorporates feedback from many community
> > members:
> > > >>> Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb,
> > > >>> Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao
> G.
> > > >>> Code reviewers on individual patches include Chris Nauroth,
> > Alejandro,
> > > >>> Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara
> Rao
> > G.
> > > >>>
> > > >>> Also thanks to Dhruba for bringing up this JIRA and thanks to
> others
> > > >> who
> > > >>> participated for discussions.
> > > >>> This vote will run for a week and close on 5/21/2014 at 06:16 pm
> IST.
> > > >>>
> > > >>> Here is my +1 to start with.
> > > >>> Regards,
> > > >>> Uma
> > > >>> (umamahesh@apache.org<ma...@apache.org>)
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Posted by Suresh Srinivas <su...@hortonworks.com>.
NN layout version bump should be fine. This means during rolling upgrade
downgrade from 2.5 NN to 2.4 NN will not be possible. Only rollback can be
supported.
HDFS-2006 adds new RPCs. The question is, does the new client that depends
on this functionality gracefully fallback when talking to old server? Also
are there any incompatibilities between NN and DN (my guess is there is
none, given this is NN feature)?
On Fri, May 16, 2014 at 5:09 PM, Andrew Wang <an...@cloudera.com>wrote:
> Hi Suresh,
>
> HDFS-2006 adds new edit log ops (no modifications to existing ops) and a
> new XAttr INode Feature needs to be serialized to the fsimage. This
> required doing a NN layout version bump, so a 2.4 NN will not be able to
> start with these edit logs or images. Thus it's incompatible.
>
> I think you're better equipped to comment on rolling upgrades, but based on
> my understanding, I think the above means we should still be able to
> rolling upgrade between 2.4 and whatever version eventually includes xattr
> support.
>
> RPC-wise, HDFS-2006 only adds new RPCs, so I don't think there are any
> concerns there.
>
> Thanks,
> Andrew
>
>
> On Fri, May 16, 2014 at 2:56 PM, Suresh Srinivas <suresh@hortonworks.com
> >wrote:
>
> > I have not looked at the development closely. With rolling upgrades
> > feature support in, are there any incompatible changes with this feature?
> >
> > Sent from phone
> >
> > > On May 16, 2014, at 10:30 AM, Chris Nauroth <cn...@hortonworks.com>
> > wrote:
> > >
> > > +1 for the merge.
> > >
> > > I've participated in ongoing design discussions and code reviews on
> > > individual patches. Yesterday, I completed a final review pass over
> the
> > > code in the feature branch, the design document, the end user
> > documentation
> > > and the test plan. It looks ready to me.
> > >
> > > Thank you to everyone who made contributions on the feature branch.
> > >
> > > Chris Nauroth
> > > Hortonworks
> > > http://hortonworks.com/
> > >
> > >
> > >
> > > On Wed, May 14, 2014 at 9:20 AM, Andrew Wang <andrew.wang@cloudera.com
> > >wrote:
> > >
> > >> +1 from me as well, I've participated in review and development of
> this
> > >> branch and think it's ready for merge.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >>
> > >> On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma
> > >> <um...@intel.com>wrote:
> > >>
> > >>> Hello HDFS Devs,
> > >>> I would like to call for a vote to merge the HDFS Extended
> Attributes
> > >>> (XAttrs) feature from the HDFS-2006 branch to the trunk.
> > >>> XAttrs are already widely supported on many operating systems,
> > >> including
> > >>> Linux, Windows, and Mac OS. This will allow storing attributes for
> HDFS
> > >>> file/directory.
> > >>> XAttr consist of a name and a value and exist in one of 4
> namespaces:
> > >>> user, trusted, security, and system. An XAttr name is prefixed with
> one
> > >> of
> > >>> these namespaces, so for example, "user.myxattr".
> > >>> Consistent with ongoing awareness of Namenode memory usage, the
> > maximum
> > >>> number and size of XAttrs on a file/directory are limited by a
> > >>> configuration parameter.
> > >>> The design document contains more details and can be found here:
> > >>
> >
> https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
> > >>> Development of this feature has been tracked in JIRA HDFS-2006:
> > >>> https://issues.apache.org/jira/browse/HDFS-2006
> > >>> All of the development work for the feature is contained in the
> > >>> "HDFS-2006" branch:
> > >>> https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
> > >>> As last tasks, we are working to support XAttrs via libhdfs, webhdfs
> as
> > >>> well as other minor improvements.
> > >>> We intend to finish those enhancements before the vote completes and
> > >>> otherwise we could move them to top-level JIRAs as they can be
> tracked
> > >>> independently. User document is also ready for this feature.
> > >>> Here the doc attached in JIRA:
> > >>
> >
> https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
> > >>> The XAttrs feature is backwards-compatible and enabled by default. A
> > >>> cluster administrator can disable it.
> > >>> Testing:
> > >>> We've developed more than 70 new tests which cover the XAttrs get,
> set
> > >>> and remove APIs through DistributedFileSystem and WebHdfsFileSystem,
> > the
> > >>> new XAttr CLI commands, HA, XAttr persistence in the fsimage and
> > related.
> > >>> Additional testing plans are documented in:
> > >>
> >
> https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
> > >>> Thanks a lot to the contributors who have helped and participated in
> > >> the
> > >>> branch development.
> > >>> Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma
> > >>> Maheswara Rao G.
> > >>> The design document incorporates feedback from many community
> members:
> > >>> Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb,
> > >>> Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
> > >>> Code reviewers on individual patches include Chris Nauroth,
> Alejandro,
> > >>> Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao
> G.
> > >>>
> > >>> Also thanks to Dhruba for bringing up this JIRA and thanks to others
> > >> who
> > >>> participated for discussions.
> > >>> This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
> > >>>
> > >>> Here is my +1 to start with.
> > >>> Regards,
> > >>> Uma
> > >>> (umamahesh@apache.org<ma...@apache.org>)
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
http://hortonworks.com/download/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Posted by Andrew Wang <an...@cloudera.com>.
Hi Suresh,
HDFS-2006 adds new edit log ops (no modifications to existing ops) and a
new XAttr INode Feature needs to be serialized to the fsimage. This
required doing a NN layout version bump, so a 2.4 NN will not be able to
start with these edit logs or images. Thus it's incompatible.
I think you're better equipped to comment on rolling upgrades, but based on
my understanding, I think the above means we should still be able to
rolling upgrade between 2.4 and whatever version eventually includes xattr
support.
RPC-wise, HDFS-2006 only adds new RPCs, so I don't think there are any
concerns there.
Thanks,
Andrew
On Fri, May 16, 2014 at 2:56 PM, Suresh Srinivas <su...@hortonworks.com>wrote:
> I have not looked at the development closely. With rolling upgrades
> feature support in, are there any incompatible changes with this feature?
>
> Sent from phone
>
> > On May 16, 2014, at 10:30 AM, Chris Nauroth <cn...@hortonworks.com>
> wrote:
> >
> > +1 for the merge.
> >
> > I've participated in ongoing design discussions and code reviews on
> > individual patches. Yesterday, I completed a final review pass over the
> > code in the feature branch, the design document, the end user
> documentation
> > and the test plan. It looks ready to me.
> >
> > Thank you to everyone who made contributions on the feature branch.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> >
> >
> > On Wed, May 14, 2014 at 9:20 AM, Andrew Wang <andrew.wang@cloudera.com
> >wrote:
> >
> >> +1 from me as well, I've participated in review and development of this
> >> branch and think it's ready for merge.
> >>
> >> Thanks,
> >> Andrew
> >>
> >>
> >> On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma
> >> <um...@intel.com>wrote:
> >>
> >>> Hello HDFS Devs,
> >>> I would like to call for a vote to merge the HDFS Extended Attributes
> >>> (XAttrs) feature from the HDFS-2006 branch to the trunk.
> >>> XAttrs are already widely supported on many operating systems,
> >> including
> >>> Linux, Windows, and Mac OS. This will allow storing attributes for HDFS
> >>> file/directory.
> >>> XAttr consist of a name and a value and exist in one of 4 namespaces:
> >>> user, trusted, security, and system. An XAttr name is prefixed with one
> >> of
> >>> these namespaces, so for example, "user.myxattr".
> >>> Consistent with ongoing awareness of Namenode memory usage, the
> maximum
> >>> number and size of XAttrs on a file/directory are limited by a
> >>> configuration parameter.
> >>> The design document contains more details and can be found here:
> >>
> https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
> >>> Development of this feature has been tracked in JIRA HDFS-2006:
> >>> https://issues.apache.org/jira/browse/HDFS-2006
> >>> All of the development work for the feature is contained in the
> >>> "HDFS-2006" branch:
> >>> https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
> >>> As last tasks, we are working to support XAttrs via libhdfs, webhdfs as
> >>> well as other minor improvements.
> >>> We intend to finish those enhancements before the vote completes and
> >>> otherwise we could move them to top-level JIRAs as they can be tracked
> >>> independently. User document is also ready for this feature.
> >>> Here the doc attached in JIRA:
> >>
> https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
> >>> The XAttrs feature is backwards-compatible and enabled by default. A
> >>> cluster administrator can disable it.
> >>> Testing:
> >>> We've developed more than 70 new tests which cover the XAttrs get, set
> >>> and remove APIs through DistributedFileSystem and WebHdfsFileSystem,
> the
> >>> new XAttr CLI commands, HA, XAttr persistence in the fsimage and
> related.
> >>> Additional testing plans are documented in:
> >>
> https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
> >>> Thanks a lot to the contributors who have helped and participated in
> >> the
> >>> branch development.
> >>> Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma
> >>> Maheswara Rao G.
> >>> The design document incorporates feedback from many community members:
> >>> Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb,
> >>> Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
> >>> Code reviewers on individual patches include Chris Nauroth, Alejandro,
> >>> Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
> >>>
> >>> Also thanks to Dhruba for bringing up this JIRA and thanks to others
> >> who
> >>> participated for discussions.
> >>> This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
> >>>
> >>> Here is my +1 to start with.
> >>> Regards,
> >>> Uma
> >>> (umamahesh@apache.org<ma...@apache.org>)
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Posted by Suresh Srinivas <su...@hortonworks.com>.
I have not looked at the development closely. With rolling upgrades feature support in, are there any incompatible changes with this feature?
Sent from phone
> On May 16, 2014, at 10:30 AM, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> +1 for the merge.
>
> I've participated in ongoing design discussions and code reviews on
> individual patches. Yesterday, I completed a final review pass over the
> code in the feature branch, the design document, the end user documentation
> and the test plan. It looks ready to me.
>
> Thank you to everyone who made contributions on the feature branch.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Wed, May 14, 2014 at 9:20 AM, Andrew Wang <an...@cloudera.com>wrote:
>
>> +1 from me as well, I've participated in review and development of this
>> branch and think it's ready for merge.
>>
>> Thanks,
>> Andrew
>>
>>
>> On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma
>> <um...@intel.com>wrote:
>>
>>> Hello HDFS Devs,
>>> I would like to call for a vote to merge the HDFS Extended Attributes
>>> (XAttrs) feature from the HDFS-2006 branch to the trunk.
>>> XAttrs are already widely supported on many operating systems,
>> including
>>> Linux, Windows, and Mac OS. This will allow storing attributes for HDFS
>>> file/directory.
>>> XAttr consist of a name and a value and exist in one of 4 namespaces:
>>> user, trusted, security, and system. An XAttr name is prefixed with one
>> of
>>> these namespaces, so for example, "user.myxattr".
>>> Consistent with ongoing awareness of Namenode memory usage, the maximum
>>> number and size of XAttrs on a file/directory are limited by a
>>> configuration parameter.
>>> The design document contains more details and can be found here:
>> https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
>>> Development of this feature has been tracked in JIRA HDFS-2006:
>>> https://issues.apache.org/jira/browse/HDFS-2006
>>> All of the development work for the feature is contained in the
>>> "HDFS-2006" branch:
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
>>> As last tasks, we are working to support XAttrs via libhdfs, webhdfs as
>>> well as other minor improvements.
>>> We intend to finish those enhancements before the vote completes and
>>> otherwise we could move them to top-level JIRAs as they can be tracked
>>> independently. User document is also ready for this feature.
>>> Here the doc attached in JIRA:
>> https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
>>> The XAttrs feature is backwards-compatible and enabled by default. A
>>> cluster administrator can disable it.
>>> Testing:
>>> We've developed more than 70 new tests which cover the XAttrs get, set
>>> and remove APIs through DistributedFileSystem and WebHdfsFileSystem, the
>>> new XAttr CLI commands, HA, XAttr persistence in the fsimage and related.
>>> Additional testing plans are documented in:
>> https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
>>> Thanks a lot to the contributors who have helped and participated in
>> the
>>> branch development.
>>> Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma
>>> Maheswara Rao G.
>>> The design document incorporates feedback from many community members:
>>> Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb,
>>> Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
>>> Code reviewers on individual patches include Chris Nauroth, Alejandro,
>>> Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
>>>
>>> Also thanks to Dhruba for bringing up this JIRA and thanks to others
>> who
>>> participated for discussions.
>>> This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
>>>
>>> Here is my +1 to start with.
>>> Regards,
>>> Uma
>>> (umamahesh@apache.org<ma...@apache.org>)
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Posted by Chris Nauroth <cn...@hortonworks.com>.
+1 for the merge.
I've participated in ongoing design discussions and code reviews on
individual patches. Yesterday, I completed a final review pass over the
code in the feature branch, the design document, the end user documentation
and the test plan. It looks ready to me.
Thank you to everyone who made contributions on the feature branch.
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Wed, May 14, 2014 at 9:20 AM, Andrew Wang <an...@cloudera.com>wrote:
> +1 from me as well, I've participated in review and development of this
> branch and think it's ready for merge.
>
> Thanks,
> Andrew
>
>
> On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma
> <um...@intel.com>wrote:
>
> > Hello HDFS Devs,
> > I would like to call for a vote to merge the HDFS Extended Attributes
> > (XAttrs) feature from the HDFS-2006 branch to the trunk.
> > XAttrs are already widely supported on many operating systems,
> including
> > Linux, Windows, and Mac OS. This will allow storing attributes for HDFS
> > file/directory.
> > XAttr consist of a name and a value and exist in one of 4 namespaces:
> > user, trusted, security, and system. An XAttr name is prefixed with one
> of
> > these namespaces, so for example, "user.myxattr".
> > Consistent with ongoing awareness of Namenode memory usage, the maximum
> > number and size of XAttrs on a file/directory are limited by a
> > configuration parameter.
> > The design document contains more details and can be found here:
> >
> https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
> > Development of this feature has been tracked in JIRA HDFS-2006:
> > https://issues.apache.org/jira/browse/HDFS-2006
> > All of the development work for the feature is contained in the
> > "HDFS-2006" branch:
> > https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
> > As last tasks, we are working to support XAttrs via libhdfs, webhdfs as
> > well as other minor improvements.
> > We intend to finish those enhancements before the vote completes and
> > otherwise we could move them to top-level JIRAs as they can be tracked
> > independently. User document is also ready for this feature.
> > Here the doc attached in JIRA:
> >
> https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
> > The XAttrs feature is backwards-compatible and enabled by default. A
> > cluster administrator can disable it.
> > Testing:
> > We've developed more than 70 new tests which cover the XAttrs get, set
> > and remove APIs through DistributedFileSystem and WebHdfsFileSystem, the
> > new XAttr CLI commands, HA, XAttr persistence in the fsimage and related.
> > Additional testing plans are documented in:
> >
> https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
> > Thanks a lot to the contributors who have helped and participated in
> the
> > branch development.
> > Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma
> > Maheswara Rao G.
> > The design document incorporates feedback from many community members:
> > Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb,
> > Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
> > Code reviewers on individual patches include Chris Nauroth, Alejandro,
> > Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
> >
> > Also thanks to Dhruba for bringing up this JIRA and thanks to others
> who
> > participated for discussions.
> > This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
> >
> > Here is my +1 to start with.
> > Regards,
> > Uma
> > (umamahesh@apache.org<ma...@apache.org>)
> >
> >
> >
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Posted by Andrew Wang <an...@cloudera.com>.
+1 from me as well, I've participated in review and development of this
branch and think it's ready for merge.
Thanks,
Andrew
On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma
<um...@intel.com>wrote:
> Hello HDFS Devs,
> I would like to call for a vote to merge the HDFS Extended Attributes
> (XAttrs) feature from the HDFS-2006 branch to the trunk.
> XAttrs are already widely supported on many operating systems, including
> Linux, Windows, and Mac OS. This will allow storing attributes for HDFS
> file/directory.
> XAttr consist of a name and a value and exist in one of 4 namespaces:
> user, trusted, security, and system. An XAttr name is prefixed with one of
> these namespaces, so for example, "user.myxattr".
> Consistent with ongoing awareness of Namenode memory usage, the maximum
> number and size of XAttrs on a file/directory are limited by a
> configuration parameter.
> The design document contains more details and can be found here:
> https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
> Development of this feature has been tracked in JIRA HDFS-2006:
> https://issues.apache.org/jira/browse/HDFS-2006
> All of the development work for the feature is contained in the
> "HDFS-2006" branch:
> https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
> As last tasks, we are working to support XAttrs via libhdfs, webhdfs as
> well as other minor improvements.
> We intend to finish those enhancements before the vote completes and
> otherwise we could move them to top-level JIRAs as they can be tracked
> independently. User document is also ready for this feature.
> Here the doc attached in JIRA:
> https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
> The XAttrs feature is backwards-compatible and enabled by default. A
> cluster administrator can disable it.
> Testing:
> We've developed more than 70 new tests which cover the XAttrs get, set
> and remove APIs through DistributedFileSystem and WebHdfsFileSystem, the
> new XAttr CLI commands, HA, XAttr persistence in the fsimage and related.
> Additional testing plans are documented in:
> https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
> Thanks a lot to the contributors who have helped and participated in the
> branch development.
> Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma
> Maheswara Rao G.
> The design document incorporates feedback from many community members:
> Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb,
> Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
> Code reviewers on individual patches include Chris Nauroth, Alejandro,
> Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
>
> Also thanks to Dhruba for bringing up this JIRA and thanks to others who
> participated for discussions.
> This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
>
> Here is my +1 to start with.
> Regards,
> Uma
> (umamahesh@apache.org<ma...@apache.org>)
>
>
>
>
RE: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to
Trunk
Posted by "Gangumalla, Uma" <um...@intel.com>.
I have merged this feature to Branch-2 now.
>From now onwards if any issues related to Xattrs, please merge them to branch-2 if needed.
I will merge the remaining jiras tomorrow which are related to Xattrs feature but handled as top level Jiras. Ex: DistCp(MAPREDUCE-5898) support etc
Regards,
Uma
-----Original Message-----
From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]
Sent: Wednesday, May 21, 2014 8:06 PM
To: hdfs-dev@hadoop.apache.org
Subject: RE: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Thanks a lot, for the great work on branch and support.
I have just completed the merge of HDFS Extended attributes branch(HDFS-2006) to trunk.
Regards,
Uma
-----Original Message-----
From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]
Sent: Wednesday, May 21, 2014 6:38 PM
To: hdfs-dev@hadoop.apache.org
Subject: RE: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Thanks a lot for participating in this vote.
With 4 +1's( from Me, Andrew Wang, Chris and Colin) and no -1, the vote has passed for the merge.
I will do the merge shortly to trunk.
Regards,
Uma
-----Original Message-----
From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]
Sent: Wednesday, May 14, 2014 6:17 PM
To: hdfs-dev@hadoop.apache.org
Subject: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Hello HDFS Devs,
I would like to call for a vote to merge the HDFS Extended Attributes (XAttrs) feature from the HDFS-2006 branch to the trunk.
XAttrs are already widely supported on many operating systems, including Linux, Windows, and Mac OS. This will allow storing attributes for HDFS file/directory.
XAttr consist of a name and a value and exist in one of 4 namespaces: user, trusted, security, and system. An XAttr name is prefixed with one of these namespaces, so for example, "user.myxattr".
Consistent with ongoing awareness of Namenode memory usage, the maximum number and size of XAttrs on a file/directory are limited by a configuration parameter.
The design document contains more details and can be found here: https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
Development of this feature has been tracked in JIRA HDFS-2006: https://issues.apache.org/jira/browse/HDFS-2006
All of the development work for the feature is contained in the "HDFS-2006" branch: https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
As last tasks, we are working to support XAttrs via libhdfs, webhdfs as well as other minor improvements.
We intend to finish those enhancements before the vote completes and otherwise we could move them to top-level JIRAs as they can be tracked independently. User document is also ready for this feature.
Here the doc attached in JIRA: https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
The XAttrs feature is backwards-compatible and enabled by default. A cluster administrator can disable it.
Testing:
We've developed more than 70 new tests which cover the XAttrs get, set and remove APIs through DistributedFileSystem and WebHdfsFileSystem, the new XAttr CLI commands, HA, XAttr persistence in the fsimage and related.
Additional testing plans are documented in: https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
Thanks a lot to the contributors who have helped and participated in the branch development.
Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma Maheswara Rao G.
The design document incorporates feedback from many community members: Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb, Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
Code reviewers on individual patches include Chris Nauroth, Alejandro, Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
Also thanks to Dhruba for bringing up this JIRA and thanks to others who participated for discussions.
This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
Here is my +1 to start with.
Regards,
Uma
(umamahesh@apache.org<ma...@apache.org>)
RE: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to
Trunk
Posted by "Gangumalla, Uma" <um...@intel.com>.
Thanks a lot, for the great work on branch and support.
I have just completed the merge of HDFS Extended attributes branch(HDFS-2006) to trunk.
Regards,
Uma
-----Original Message-----
From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]
Sent: Wednesday, May 21, 2014 6:38 PM
To: hdfs-dev@hadoop.apache.org
Subject: RE: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Thanks a lot for participating in this vote.
With 4 +1's( from Me, Andrew Wang, Chris and Colin) and no -1, the vote has passed for the merge.
I will do the merge shortly to trunk.
Regards,
Uma
-----Original Message-----
From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]
Sent: Wednesday, May 14, 2014 6:17 PM
To: hdfs-dev@hadoop.apache.org
Subject: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Hello HDFS Devs,
I would like to call for a vote to merge the HDFS Extended Attributes (XAttrs) feature from the HDFS-2006 branch to the trunk.
XAttrs are already widely supported on many operating systems, including Linux, Windows, and Mac OS. This will allow storing attributes for HDFS file/directory.
XAttr consist of a name and a value and exist in one of 4 namespaces: user, trusted, security, and system. An XAttr name is prefixed with one of these namespaces, so for example, "user.myxattr".
Consistent with ongoing awareness of Namenode memory usage, the maximum number and size of XAttrs on a file/directory are limited by a configuration parameter.
The design document contains more details and can be found here: https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
Development of this feature has been tracked in JIRA HDFS-2006: https://issues.apache.org/jira/browse/HDFS-2006
All of the development work for the feature is contained in the "HDFS-2006" branch: https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
As last tasks, we are working to support XAttrs via libhdfs, webhdfs as well as other minor improvements.
We intend to finish those enhancements before the vote completes and otherwise we could move them to top-level JIRAs as they can be tracked independently. User document is also ready for this feature.
Here the doc attached in JIRA: https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
The XAttrs feature is backwards-compatible and enabled by default. A cluster administrator can disable it.
Testing:
We've developed more than 70 new tests which cover the XAttrs get, set and remove APIs through DistributedFileSystem and WebHdfsFileSystem, the new XAttr CLI commands, HA, XAttr persistence in the fsimage and related.
Additional testing plans are documented in: https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
Thanks a lot to the contributors who have helped and participated in the branch development.
Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma Maheswara Rao G.
The design document incorporates feedback from many community members: Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb, Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
Code reviewers on individual patches include Chris Nauroth, Alejandro, Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
Also thanks to Dhruba for bringing up this JIRA and thanks to others who participated for discussions.
This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
Here is my +1 to start with.
Regards,
Uma
(umamahesh@apache.org<ma...@apache.org>)
RE: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to
Trunk
Posted by "Gangumalla, Uma" <um...@intel.com>.
Thanks a lot for participating in this vote.
With 4 +1's( from Me, Andrew Wang, Chris and Colin) and no -1, the vote has passed for the merge.
I will do the merge shortly to trunk.
Regards,
Uma
-----Original Message-----
From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]
Sent: Wednesday, May 14, 2014 6:17 PM
To: hdfs-dev@hadoop.apache.org
Subject: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Hello HDFS Devs,
I would like to call for a vote to merge the HDFS Extended Attributes (XAttrs) feature from the HDFS-2006 branch to the trunk.
XAttrs are already widely supported on many operating systems, including Linux, Windows, and Mac OS. This will allow storing attributes for HDFS file/directory.
XAttr consist of a name and a value and exist in one of 4 namespaces: user, trusted, security, and system. An XAttr name is prefixed with one of these namespaces, so for example, "user.myxattr".
Consistent with ongoing awareness of Namenode memory usage, the maximum number and size of XAttrs on a file/directory are limited by a configuration parameter.
The design document contains more details and can be found here: https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
Development of this feature has been tracked in JIRA HDFS-2006: https://issues.apache.org/jira/browse/HDFS-2006
All of the development work for the feature is contained in the "HDFS-2006" branch: https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
As last tasks, we are working to support XAttrs via libhdfs, webhdfs as well as other minor improvements.
We intend to finish those enhancements before the vote completes and otherwise we could move them to top-level JIRAs as they can be tracked independently. User document is also ready for this feature.
Here the doc attached in JIRA: https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
The XAttrs feature is backwards-compatible and enabled by default. A cluster administrator can disable it.
Testing:
We've developed more than 70 new tests which cover the XAttrs get, set and remove APIs through DistributedFileSystem and WebHdfsFileSystem, the new XAttr CLI commands, HA, XAttr persistence in the fsimage and related.
Additional testing plans are documented in: https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
Thanks a lot to the contributors who have helped and participated in the branch development.
Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma Maheswara Rao G.
The design document incorporates feedback from many community members: Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb, Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
Code reviewers on individual patches include Chris Nauroth, Alejandro, Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
Also thanks to Dhruba for bringing up this JIRA and thanks to others who participated for discussions.
This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
Here is my +1 to start with.
Regards,
Uma
(umamahesh@apache.org<ma...@apache.org>)