You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@twill.apache.org by Steve Loughran <st...@hortonworks.com> on 2014/10/08 22:52:59 UTC
FYI, yarn registry checked in, includes some twill code
I'm just letting everyone know the core YARN-913 registry is checked in,
with the goal of a ZK-based registry for YARN apps. There's security
support too: the RM will create a zknode for a user with the right
permissions for that user and system accounts only, user apps are free to
register whatever they want underneath.
This patch actually contains a bit of twill code -your in-VM zookeeper
service was lifted, wrapped in a YARN service and will -once the remaining
patches go in- be integrated with the MiniYARNCluster.
https://git-wip-us.apache.org/repos/asf?p=hadoop.git;a=blob;f=hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java;h=3fa0c1920dd150ec23995f9b8e714d81633a9f74;hb=HEAD
Can I therefore thank the team for your contribution to the hadoop codebase
—and I hope to see you using the registry itself at some point in the future
-steve
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: FYI, yarn registry checked in, includes some twill code
Posted by Steve Loughran <st...@hortonworks.com>.
that's a good question:
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/registry/registry-security.md
Once a patch goes into the RM, it will set up the path so that each user
gets their path /users/${shortname}/ writeable only by them and the
config-defined system accounts, eg. sasl:stevel@EXAMPLE, sasl:mapred@EXAMPLE
apps can write under that, using SASL auth. There some support for allowing
a client to add digest auth ACLs so that you could delegate access to a bit
of your own tree, e.g /users/stevel/myapp/clients could be given
sasl:steve@EXAMPLE, digest:55ff44. If the id+pass for the digest is
(securely) passed down, then bits of an app without ZK tickets can still
work that bit of the registry. My goal there was to allow long-lived
services to avoid the token expiry problem. I don't know how well it would
work in practise though
All of the registry is world readable: if you want to share secrets, don't
do it directly in the registry.
On 8 October 2014 15:00, Gary Helmling <gh...@gmail.com> wrote:
> Thanks for the update, Steve. Glad to hear that the Twill code could
> help make this happen in YARN!
>
> We'll have to study up on this. I'm particularly interested in the
> security implementation. Does the RM mediate the ZK access for
> applications, or do applications directly register under their parent
> znode?
>
> On Wed, Oct 8, 2014 at 1:52 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
> > I'm just letting everyone know the core YARN-913 registry is checked in,
> > with the goal of a ZK-based registry for YARN apps. There's security
> > support too: the RM will create a zknode for a user with the right
> > permissions for that user and system accounts only, user apps are free to
> > register whatever they want underneath.
> >
> > This patch actually contains a bit of twill code -your in-VM zookeeper
> > service was lifted, wrapped in a YARN service and will -once the
> remaining
> > patches go in- be integrated with the MiniYARNCluster.
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=hadoop.git;a=blob;f=hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java;h=3fa0c1920dd150ec23995f9b8e714d81633a9f74;hb=HEAD
> >
> > Can I therefore thank the team for your contribution to the hadoop
> codebase
> > —and I hope to see you using the registry itself at some point in the
> future
> >
> > -steve
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: FYI, yarn registry checked in, includes some twill code
Posted by Gary Helmling <gh...@gmail.com>.
Thanks for the update, Steve. Glad to hear that the Twill code could
help make this happen in YARN!
We'll have to study up on this. I'm particularly interested in the
security implementation. Does the RM mediate the ZK access for
applications, or do applications directly register under their parent
znode?
On Wed, Oct 8, 2014 at 1:52 PM, Steve Loughran <st...@hortonworks.com> wrote:
> I'm just letting everyone know the core YARN-913 registry is checked in,
> with the goal of a ZK-based registry for YARN apps. There's security
> support too: the RM will create a zknode for a user with the right
> permissions for that user and system accounts only, user apps are free to
> register whatever they want underneath.
>
> This patch actually contains a bit of twill code -your in-VM zookeeper
> service was lifted, wrapped in a YARN service and will -once the remaining
> patches go in- be integrated with the MiniYARNCluster.
>
> https://git-wip-us.apache.org/repos/asf?p=hadoop.git;a=blob;f=hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java;h=3fa0c1920dd150ec23995f9b8e714d81633a9f74;hb=HEAD
>
> Can I therefore thank the team for your contribution to the hadoop codebase
> —and I hope to see you using the registry itself at some point in the future
>
> -steve
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.