You are viewing a plain text version of this content. The canonical link for it is here.
Posted to kerby@directory.apache.org by "Li, Jiajia" <ji...@intel.com> on 2017/11/24 03:30:20 UTC
[DISCUSS] Merge HAS to Apache Kerby
Hi all,
I would like to post a proposal about merging a new project HAS (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel and Alibaba, it is a solution to support the authentication of open source big data ecosystem in cloud computing platforms. I've created a new branch "has-project" in Kerby, HAS is under "has" folder. Please look at https://github.com/apache/directory-kerby/tree/has-project/has for details.
Background and motivation:
At present, the open source big data ecosystems (Hadoop/Spark) only has the built-in Kerberos support on the security authentication. HAS aims to build a standalone authentication service for the big data ecosystem that simplifies the support of Kerberos and allows to use more authentication methods.
Targets users:
HAS supports various authentication mechanisms other than just Kerberos, and it provides a new authentication mechanism can be easy customized and plugin with existing user authentication and authorization system, and security admins won't have to migrate and sync up their user accounts to Kerberos back and forth.
Architecture & Design:
HAS provides a new authentication mechanism ("Kerberos-based token authentication"), depending on the "TokenPreauth" provided by Apache Kerby. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
Features:
1. Provides new authentication mechanism plugin APIs to customize and plugin with existing user authentication and authorization system. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
2. Provides lots of REST APIs and facility tools to simplify the support of Kerberos. Kerberos is essentially a protocol, or secure channel, doesn't have to be that complex to users. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/rest-api.md for details.
3. Provides MySQL backend for High Availability. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/mysql-backend.md for details.
4. New authentication mechanism now supports most of the components of open source big data ecosystem with little or no changes to components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please look at https://github.com/apache/directory-kerby/tree/has-project/has/supports for details.
Practice
This solution has been deployed in Alibaba Cloud E-MapReduce production.
Why to merge?
HAS provides a complete Hadoop/Spark authentication framework and solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it more solid and stronger. And if HAS can be merged to Apache Kerby, community will help HAS grow faster and users can more easily using this solution in their own production. We have two suggestions about how to merge:
- Option1:
Create a standalone module "kerby-has", putting HAS project under this module.
- Option2:
Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
Contributors:
Jiajia, Li (Intel)
Lin, Zeng (Intel)
Zhiqiang, Zhang (Intel)
Kai, Zheng (Intel)
Wei, Wu (Alibaba)
Jun, Song (Alibaba)
Long, Cao (Alibaba)
Zhenyuan, Wei (Alibaba)
Your review efforts are truly appreciated, please feel free to provide us your feedback.
Regards,
Jiajia
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
+1.
Colm.
On Fri, Dec 15, 2017 at 3:52 AM, 郑锴(铁杰) <zh...@alibaba-inc.com> wrote:
> Thanks Jiajia for the update and major progress. It would be good to see
> HAS feature consolidated into Kerby KDC offering.
>
> Regards,
> Kai
>
> ------------------------------------------------------------------
> 发件人：Li, Jiajia <ji...@intel.com>
> 发送时间：2017年12月15日(星期五) 10:38
> 收件人：Apache Directory Developers List <de...@directory.apache.org>;
> coheigea@apache.org <co...@apache.org>
> 抄 送：kerby@directory.apache.org <ke...@directory.apache.org>
> 主 题：RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi all, <#m_8069559770700476617_this>
>
>
>
> Status update:
>
>
>
> 1. I've checked both Intel and Alibaba have signed the CCLA.
>
> 2. We have fixed dependency issues: mysql JDBC driver is replaced with
> Drizzle JDBC and some dependencies cannot find a license have been removed.
>
> 3. If there are no more questions, we will start to merge under the master
> JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to
> review the patches.
>
>
>
> Thanks,
>
> Jiajia
>
>
>
> <#m_8069559770700476617_this>*From:* Colm O hEigeartaigh [mailto:
> coheigea@apache.org]
> *Sent:* Wednesday, December 6, 2017 11:23 PM
> *To:* Li, Jiajia <ji...@intel.com>
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>;
> kerby@directory.apache.org
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Hi Jiajia,
>
> Perhaps you could get one of the Alibaba contributors to mail "
> secretary@apache.org" and ask if there is a CCLA on record?
>
> Colm.
>
>
>
> On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> I think Intel have provided the CCLA when contributing Kerby to Apache.
> But I'm not sure whether Alibaba already provided, is there one place we
> could check it?
>
>
>
> Thanks,
>
> Jiajia
>
> <#m_8069559770700476617_this>
>
> <#m_8069559770700476617_this>*From:* Colm O hEigeartaigh [mailto:
> coheigea@apache.org]
> *Sent:* Tuesday, December 5, 2017 1:50 AM
> *To:* kerby@directory.apache.org
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Do we have both CCLAs filed for Intel and Alibaba?
>
> Colm.
>
>
>
> On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
>
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
>
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
+1.
Colm.
On Fri, Dec 15, 2017 at 3:52 AM, 郑锴(铁杰) <zh...@alibaba-inc.com> wrote:
> Thanks Jiajia for the update and major progress. It would be good to see
> HAS feature consolidated into Kerby KDC offering.
>
> Regards,
> Kai
>
> ------------------------------------------------------------------
> 发件人：Li, Jiajia <ji...@intel.com>
> 发送时间：2017年12月15日(星期五) 10:38
> 收件人：Apache Directory Developers List <de...@directory.apache.org>;
> coheigea@apache.org <co...@apache.org>
> 抄 送：kerby@directory.apache.org <ke...@directory.apache.org>
> 主 题：RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi all, <#m_8069559770700476617_this>
>
>
>
> Status update:
>
>
>
> 1. I've checked both Intel and Alibaba have signed the CCLA.
>
> 2. We have fixed dependency issues: mysql JDBC driver is replaced with
> Drizzle JDBC and some dependencies cannot find a license have been removed.
>
> 3. If there are no more questions, we will start to merge under the master
> JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to
> review the patches.
>
>
>
> Thanks,
>
> Jiajia
>
>
>
> <#m_8069559770700476617_this>*From:* Colm O hEigeartaigh [mailto:
> coheigea@apache.org]
> *Sent:* Wednesday, December 6, 2017 11:23 PM
> *To:* Li, Jiajia <ji...@intel.com>
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>;
> kerby@directory.apache.org
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Hi Jiajia,
>
> Perhaps you could get one of the Alibaba contributors to mail "
> secretary@apache.org" and ask if there is a CCLA on record?
>
> Colm.
>
>
>
> On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> I think Intel have provided the CCLA when contributing Kerby to Apache.
> But I'm not sure whether Alibaba already provided, is there one place we
> could check it?
>
>
>
> Thanks,
>
> Jiajia
>
> <#m_8069559770700476617_this>
>
> <#m_8069559770700476617_this>*From:* Colm O hEigeartaigh [mailto:
> coheigea@apache.org]
> *Sent:* Tuesday, December 5, 2017 1:50 AM
> *To:* kerby@directory.apache.org
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Do we have both CCLAs filed for Intel and Alibaba?
>
> Colm.
>
>
>
> On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
>
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
>
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
回复：[DISCUSS] Merge HAS to Apache Kerby
Posted by "郑锴(铁杰)" <zh...@alibaba-inc.com>.
Thanks Jiajia for the update and major progress. It would be good to see HAS feature consolidated into Kerby KDC offering.
Regards,Kai
------------------------------------------------------------------发件人：Li, Jiajia <ji...@intel.com>发送时间：2017年12月15日(星期五) 10:38收件人：Apache Directory Developers List <de...@directory.apache.org>; coheigea@apache.org <co...@apache.org>抄 送：kerby@directory.apache.org <ke...@directory.apache.org>主 题：RE: [DISCUSS] Merge HAS to Apache Kerby
Hi all,
Status update:
1. I've checked both Intel and Alibaba have signed the CCLA.
2. We have fixed dependency issues: mysql JDBC driver is replaced with Drizzle JDBC and some dependencies cannot find a license have been removed.
3. If there are no more questions, we will start to merge under the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to review the patches.
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, December 6, 2017 11:23 PM
To: Li, Jiajia <ji...@intel.com>
Cc: Apache Directory Developers List <de...@directory.apache.org>; kerby@directory.apache.org
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "secretary@apache.org" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba
already provided, is there one place we could check it?
Thanks,
Jiajia
From:
Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to
secretary@apache.org and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org
Cc: kerby@directory.apache.org; Apache Directory Developers List <de...@directory.apache.org>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example,
the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next
step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate
tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and
server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive
some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP
that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug
in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> >
https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> >
https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
回复：[DISCUSS] Merge HAS to Apache Kerby
Posted by "郑锴(铁杰)" <zh...@alibaba-inc.com>.
Thanks Jiajia for the update and major progress. It would be good to see HAS feature consolidated into Kerby KDC offering.
Regards,Kai
------------------------------------------------------------------发件人：Li, Jiajia <ji...@intel.com>发送时间：2017年12月15日(星期五) 10:38收件人：Apache Directory Developers List <de...@directory.apache.org>; coheigea@apache.org <co...@apache.org>抄 送：kerby@directory.apache.org <ke...@directory.apache.org>主 题：RE: [DISCUSS] Merge HAS to Apache Kerby
Hi all,
Status update:
1. I've checked both Intel and Alibaba have signed the CCLA.
2. We have fixed dependency issues: mysql JDBC driver is replaced with Drizzle JDBC and some dependencies cannot find a license have been removed.
3. If there are no more questions, we will start to merge under the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to review the patches.
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, December 6, 2017 11:23 PM
To: Li, Jiajia <ji...@intel.com>
Cc: Apache Directory Developers List <de...@directory.apache.org>; kerby@directory.apache.org
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "secretary@apache.org" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba
already provided, is there one place we could check it?
Thanks,
Jiajia
From:
Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to
secretary@apache.org and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org
Cc: kerby@directory.apache.org; Apache Directory Developers List <de...@directory.apache.org>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example,
the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next
step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate
tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and
server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive
some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP
that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug
in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> >
https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> >
https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Hi all,
Status update:
1. I've checked both Intel and Alibaba have signed the CCLA.
2. We have fixed dependency issues: mysql JDBC driver is replaced with Drizzle JDBC and some dependencies cannot find a license have been removed.
3. If there are no more questions, we will start to merge under the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to review the patches.
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, December 6, 2017 11:23 PM
To: Li, Jiajia <ji...@intel.com>
Cc: Apache Directory Developers List <de...@directory.apache.org>; kerby@directory.apache.org
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "secretary@apache.org<ma...@apache.org>" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com>> wrote:
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba already provided, is there one place we could check it?
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org<ma...@directory.apache.org>
Cc: Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com>> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to secretary@apache.org<ma...@apache.org> and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com<ma...@intel.com>]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org<ma...@apache.org>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md><http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md><http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Hi all,
Status update:
1. I've checked both Intel and Alibaba have signed the CCLA.
2. We have fixed dependency issues: mysql JDBC driver is replaced with Drizzle JDBC and some dependencies cannot find a license have been removed.
3. If there are no more questions, we will start to merge under the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to review the patches.
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, December 6, 2017 11:23 PM
To: Li, Jiajia <ji...@intel.com>
Cc: Apache Directory Developers List <de...@directory.apache.org>; kerby@directory.apache.org
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "secretary@apache.org<ma...@apache.org>" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com>> wrote:
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba already provided, is there one place we could check it?
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org<ma...@directory.apache.org>
Cc: Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com>> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to secretary@apache.org<ma...@apache.org> and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com<ma...@intel.com>]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org<ma...@apache.org>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md><http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md><http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "
secretary@apache.org" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
> I think Intel have provided the CCLA when contributing Kerby to Apache.
> But I'm not sure whether Alibaba already provided, is there one place we
> could check it?
>
>
>
> Thanks,
>
> Jiajia
>
>
>
> *From:* Colm O hEigeartaigh [mailto:coheigea@apache.org]
> *Sent:* Tuesday, December 5, 2017 1:50 AM
> *To:* kerby@directory.apache.org
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Do we have both CCLAs filed for Intel and Alibaba?
>
> Colm.
>
>
>
> On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
>
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "
secretary@apache.org" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
> I think Intel have provided the CCLA when contributing Kerby to Apache.
> But I'm not sure whether Alibaba already provided, is there one place we
> could check it?
>
>
>
> Thanks,
>
> Jiajia
>
>
>
> *From:* Colm O hEigeartaigh [mailto:coheigea@apache.org]
> *Sent:* Tuesday, December 5, 2017 1:50 AM
> *To:* kerby@directory.apache.org
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Do we have both CCLAs filed for Intel and Alibaba?
>
> Colm.
>
>
>
> On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
>
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba already provided, is there one place we could check it?
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com>> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to secretary@apache.org<ma...@apache.org> and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com<ma...@intel.com>]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org<ma...@apache.org>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md><http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md><http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba already provided, is there one place we could check it?
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com>> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to secretary@apache.org<ma...@apache.org> and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com<ma...@intel.com>]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org<ma...@apache.org>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md><http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md><http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Hi all,
Here with some status update, now all the contributors have provided the ICLA to secretary@apache.org and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org
Cc: kerby@directory.apache.org; Apache Directory Developers List <de...@directory.apache.org>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Hi Colm,
> I still think we should make it optional (by configuration) whether the users need to be in the backend or not. If it is optional we can support authentication using tokens from other IdPs. A more advanced use-case is to translate identities from a token issued by an IdP in another realm to the identities stored in the backend.
It's a good idea to make it optional, that will let it be more lightweight and scalable. Just with one minor issue, if there is no backend, how to store the krbtgt/REALM@REALM, this principal is used for issuing TGT(ticket granting ticket), and now is created by kdcinit tool.
> Yep that would be cool. I can help out with this as I'm a committer on Apache CXF. Here's a blog post I wrote on the REST interface of the STS:
> http://coheigea.blogspot.ie/2016/06/a-new-rest-interface-for-apache-cxf.html
> This is something we can look at in the future anyway once the code is merged.
It's really great if you could help with this, that will make this solution more powerful.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, December 5, 2017 1:55 AM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
Answers inline.
On Thu, Nov 30, 2017 at 5:38 AM, Li, Jiajia <ji...@intel.com> wrote:
>
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting
> the TGT, next step is to get SGT(Ticket for Service), in this step,
> the service principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead
> of service level, services are still strongly authenticated by
> Kerberos, they through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential
> cache to manage the cluster, for the compatibility.
>
I still think we should make it optional (by configuration) whether the users need to be in the backend or not. If it is optional we can support authentication using tokens from other IdPs. A more advanced use-case is to translate identities from a token issued by an IdP in another realm to the identities stored in the backend.
>
> I do not know much about SecurityTokenService, from your introduction,
> I think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the
> client and server plugins for STS, then using STS in HAS framework.
> Please correct me if I'm wrong.
>
Yep that would be cool. I can help out with this as I'm a committer on Apache CXF. Here's a blog post I wrote on the REST interface of the STS:
http://coheigea.blogspot.ie/2016/06/a-new-rest-interface-for-apache-cxf.html
This is something we can look at in the future anyway once the code is merged.
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby,
> using AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide
> some useful interfaces, such as: config Kerby KDC, manage the Kerby
> backend, export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all
> included in HasServer(we can rename it after merging), we could also
> think the webserver is one part of Kerby KDC, we using the webserver
> for KDC to receive some requests from HTTPs client.
>
Yes +1 from me on merging to Kerby, once the legal stuff is sorted out.
Colm.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
> To: Li, Jiajia <ji...@intel.com>
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend.
> For example, the token might be issued by an IdP that HAS "trusts",
> where the IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the
> tokens that are supported. It might be worth exploring if the
> functionality of HAS could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>;
> Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby Thanks Jiajia, that
> was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user
> supposed to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently,
> we have to use a token armor cache. In HAS, when it queries Kerby to
> get a TGT using the verified AuthToken, is this just an "internal"
> call so we can avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby
> versions with just a few changes. Can the changes be rolled into Kerby
> to prevent code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia
> <jiajia.li@intel.com<mailto:ji ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though
> > > based on
> > the README. Could you describe how it works from a really basic
> > point of view for say a simple Hadoop client? Normally I just have
> > to use "kinit" to get a kerberos ticket and then I am authenticated
> > to invoke on HDFS. How does HAS work differently? Where does the
> > token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the
> > cmd "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some
> > user info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module
> > from others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in
> > an AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through
> > HTTPS; 6. After HAS server receives the message, it will call the
> > server plugin, server plugin will verify the user info in AuthToken,
> > the following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module
> > from others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws
> > HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There
> > are two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos
> > back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any
> > feedback from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based
> > on the README. Could you describe how it works from a really basic
> > point of view for say a simple Hadoop client? Normally I just have
> > to use "kinit" to get a kerberos ticket and then I am authenticated
> > to invoke on HDFS. How does HAS work differently? Where does the
> > token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by
> > > Intel and Alibaba, it is a solution to support the authentication
> > > of open source big data ecosystem in cloud computing platforms.
> > > I've created a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark)
> > > only has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows
> > > to use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be
> > > easy customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate
> > > and sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by
> > > Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark....
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/sup
> > > po
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC,
> > > make it more solid and stronger. And if HAS can be merged to
> > > Apache Kerby, community will help HAS grow faster and users can
> > > more easily using this solution in their own production. We have
> > > two suggestions about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Hi Colm,
> I still think we should make it optional (by configuration) whether the users need to be in the backend or not. If it is optional we can support authentication using tokens from other IdPs. A more advanced use-case is to translate identities from a token issued by an IdP in another realm to the identities stored in the backend.
It's a good idea to make it optional, that will let it be more lightweight and scalable. Just with one minor issue, if there is no backend, how to store the krbtgt/REALM@REALM, this principal is used for issuing TGT(ticket granting ticket), and now is created by kdcinit tool.
> Yep that would be cool. I can help out with this as I'm a committer on Apache CXF. Here's a blog post I wrote on the REST interface of the STS:
> http://coheigea.blogspot.ie/2016/06/a-new-rest-interface-for-apache-cxf.html
> This is something we can look at in the future anyway once the code is merged.
It's really great if you could help with this, that will make this solution more powerful.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, December 5, 2017 1:55 AM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
Answers inline.
On Thu, Nov 30, 2017 at 5:38 AM, Li, Jiajia <ji...@intel.com> wrote:
>
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting
> the TGT, next step is to get SGT(Ticket for Service), in this step,
> the service principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead
> of service level, services are still strongly authenticated by
> Kerberos, they through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential
> cache to manage the cluster, for the compatibility.
>
I still think we should make it optional (by configuration) whether the users need to be in the backend or not. If it is optional we can support authentication using tokens from other IdPs. A more advanced use-case is to translate identities from a token issued by an IdP in another realm to the identities stored in the backend.
>
> I do not know much about SecurityTokenService, from your introduction,
> I think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the
> client and server plugins for STS, then using STS in HAS framework.
> Please correct me if I'm wrong.
>
Yep that would be cool. I can help out with this as I'm a committer on Apache CXF. Here's a blog post I wrote on the REST interface of the STS:
http://coheigea.blogspot.ie/2016/06/a-new-rest-interface-for-apache-cxf.html
This is something we can look at in the future anyway once the code is merged.
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby,
> using AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide
> some useful interfaces, such as: config Kerby KDC, manage the Kerby
> backend, export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all
> included in HasServer(we can rename it after merging), we could also
> think the webserver is one part of Kerby KDC, we using the webserver
> for KDC to receive some requests from HTTPs client.
>
Yes +1 from me on merging to Kerby, once the legal stuff is sorted out.
Colm.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
> To: Li, Jiajia <ji...@intel.com>
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend.
> For example, the token might be issued by an IdP that HAS "trusts",
> where the IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the
> tokens that are supported. It might be worth exploring if the
> functionality of HAS could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>;
> Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby Thanks Jiajia, that
> was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user
> supposed to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently,
> we have to use a token armor cache. In HAS, when it queries Kerby to
> get a TGT using the verified AuthToken, is this just an "internal"
> call so we can avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby
> versions with just a few changes. Can the changes be rolled into Kerby
> to prevent code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia
> <jiajia.li@intel.com<mailto:ji ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though
> > > based on
> > the README. Could you describe how it works from a really basic
> > point of view for say a simple Hadoop client? Normally I just have
> > to use "kinit" to get a kerberos ticket and then I am authenticated
> > to invoke on HDFS. How does HAS work differently? Where does the
> > token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the
> > cmd "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some
> > user info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module
> > from others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in
> > an AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through
> > HTTPS; 6. After HAS server receives the message, it will call the
> > server plugin, server plugin will verify the user info in AuthToken,
> > the following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module
> > from others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws
> > HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There
> > are two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos
> > back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any
> > feedback from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based
> > on the README. Could you describe how it works from a really basic
> > point of view for say a simple Hadoop client? Normally I just have
> > to use "kinit" to get a kerberos ticket and then I am authenticated
> > to invoke on HDFS. How does HAS work differently? Where does the
> > token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by
> > > Intel and Alibaba, it is a solution to support the authentication
> > > of open source big data ecosystem in cloud computing platforms.
> > > I've created a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark)
> > > only has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows
> > > to use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be
> > > easy customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate
> > > and sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by
> > > Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark....
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/sup
> > > po
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC,
> > > make it more solid and stronger. And if HAS can be merged to
> > > Apache Kerby, community will help HAS grow faster and users can
> > > more easily using this solution in their own production. We have
> > > two suggestions about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jiajia,
Answers inline.
On Thu, Nov 30, 2017 at 5:38 AM, Li, Jiajia <ji...@intel.com> wrote:
>
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
I still think we should make it optional (by configuration) whether the
users need to be in the backend or not. If it is optional we can support
authentication using tokens from other IdPs. A more advanced use-case is to
translate identities from a token issued by an IdP in another realm to the
identities stored in the backend.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
Yep that would be cool. I can help out with this as I'm a committer on
Apache CXF. Here's a blog post I wrote on the REST interface of the STS:
http://coheigea.blogspot.ie/2016/06/a-new-rest-interface-for-apache-cxf.html
This is something we can look at in the future anyway once the code is
merged.
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
Yes +1 from me on merging to Kerby, once the legal stuff is sorted out.
Colm.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
> To: Li, Jiajia <ji...@intel.com>
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jiajia,
Answers inline.
On Thu, Nov 30, 2017 at 5:38 AM, Li, Jiajia <ji...@intel.com> wrote:
>
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
I still think we should make it optional (by configuration) whether the
users need to be in the backend or not. If it is optional we can support
authentication using tokens from other IdPs. A more advanced use-case is to
translate identities from a token issued by an IdP in another realm to the
identities stored in the backend.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
Yep that would be cool. I can help out with this as I'm a committer on
Apache CXF. Here's a blog post I wrote on the REST interface of the STS:
http://coheigea.blogspot.ie/2016/06/a-new-rest-interface-for-apache-cxf.html
This is something we can look at in the future anyway once the code is
merged.
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
Yes +1 from me on merging to Kerby, once the legal stuff is sorted out.
Colm.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
> To: Li, Jiajia <ji...@intel.com>
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Hi all,
Here with some status update, now all the contributors have provided the ICLA to secretary@apache.org and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org
Cc: kerby@directory.apache.org; Apache Directory Developers List <de...@directory.apache.org>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>
Cc: kerby@directory.apache.org; Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>
Cc: kerby@directory.apache.org; Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jiajia,
What I meant with the point about the backend, is that it should be
configurable whether to just trust the signature of the presented auth
token as sufficient validation, without requiring any MySQL backend. For
example, the token might be issued by an IdP that HAS "trusts", where the
IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a
SecurityTokenService (STS). Apache CXF ships with a STS that I am very
familiar with. It is a web application that supports a SOAP and REST
interface to issue, validate tokens etc, where you can "plug in" the tokens
that are supported. It might be worth exploring if the functionality of HAS
could be integrated with the CXF STS.
Colm.
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org
> > Cc: Apache Directory Developers List <de...@directory.apache.org>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jiajia,
What I meant with the point about the backend, is that it should be
configurable whether to just trust the signature of the presented auth
token as sufficient validation, without requiring any MySQL backend. For
example, the token might be issued by an IdP that HAS "trusts", where the
IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a
SecurityTokenService (STS). Apache CXF ships with a STS that I am very
familiar with. It is a web application that supports a SOAP and REST
interface to issue, validate tokens etc, where you can "plug in" the tokens
that are supported. It might be worth exploring if the functionality of HAS
could be integrated with the CXF STS.
Colm.
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org
> > Cc: Apache Directory Developers List <de...@directory.apache.org>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Thanks Colm for taking time to review.
> There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
Users should implement their own plugins, they could customize their own plugins, and we plan to provide the default implementation as an example in the future work.
> If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
You are right, in HAS, AuthToken and TGT are transfered by HTTPS between client and server, so we skip the step using the armor cache.
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
The MySQL in design is not used as the backend, it's an example of existing user authentication system.
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Sure, we will merge these tools.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>
Cc: kerby@directory.apache.org; Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org
> Cc: Apache Directory Developers List <de...@directory.apache.org>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Thanks Colm for taking time to review.
> There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
Users should implement their own plugins, they could customize their own plugins, and we plan to provide the default implementation as an example in the future work.
> If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
You are right, in HAS, AuthToken and TGT are transfered by HTTPS between client and server, so we skip the step using the armor cache.
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
The MySQL in design is not used as the backend, it's an example of existing user authentication system.
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Sure, we will merge these tools.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>
Cc: kerby@directory.apache.org; Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org
> Cc: Apache Directory Developers List <de...@directory.apache.org>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed
them). Is the plan to provide some later on, or is the user supposed to
implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we
have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
using the verified AuthToken, is this just an "internal" call so we can
avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just
accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
with just a few changes. Can the changes be rolled into Kerby to prevent
code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point of
> view for say a simple Hadoop client? Normally I just have to use "kinit" to
> get a kerberos ticket and then I am authenticated to invoke on HDFS. How
> does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient", https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the right
> client plugin, the client plugin will collect and add some user info to
> "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken,
> // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server plugin,
> server plugin will verify the user info in AuthToken, the following is the
> server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results wrapped
> in an "AuthToken",
> // will be used to exchange a Kerberos ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication system,
> server plugin will return the verified "AuthToken" to Kerby KDC
> 8. Kerby KDC will issue the TGT ticket using the TokenPreauth, then send
> the TGT to HasClient through HTTPS
> 9. Now user login successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are two
> important benefits:
> 1. The user's principal may not be in the backend, security admins won't
> have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org
> Cc: Apache Directory Developers List <de...@directory.apache.org>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on the
> README. Could you describe how it works from a really basic point of view
> for say a simple Hadoop client? Normally I just have to use "kinit" to get
> a kerberos ticket and then I am authenticated to invoke on HDFS. How does
> HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created a
> > new branch "has-project" in Kerby, HAS is under "has" folder. Please
> > look at https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication. HAS
> > aims to build a standalone authentication service for the big data
> > ecosystem that simplifies the support of Kerberos and allows to use
> > more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/support
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it
> > more solid and stronger. And if HAS can be merged to Apache Kerby,
> > community will help HAS grow faster and users can more easily using
> > this solution in their own production. We have two suggestions about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under this
> > module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to provide
> > us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed
them). Is the plan to provide some later on, or is the user supposed to
implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we
have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
using the verified AuthToken, is this just an "internal" call so we can
avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just
accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
with just a few changes. Can the changes be rolled into Kerby to prevent
code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point of
> view for say a simple Hadoop client? Normally I just have to use "kinit" to
> get a kerberos ticket and then I am authenticated to invoke on HDFS. How
> does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient", https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the right
> client plugin, the client plugin will collect and add some user info to
> "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken,
> // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server plugin,
> server plugin will verify the user info in AuthToken, the following is the
> server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results wrapped
> in an "AuthToken",
> // will be used to exchange a Kerberos ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication system,
> server plugin will return the verified "AuthToken" to Kerby KDC
> 8. Kerby KDC will issue the TGT ticket using the TokenPreauth, then send
> the TGT to HasClient through HTTPS
> 9. Now user login successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are two
> important benefits:
> 1. The user's principal may not be in the backend, security admins won't
> have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org
> Cc: Apache Directory Developers List <de...@directory.apache.org>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on the
> README. Could you describe how it works from a really basic point of view
> for say a simple Hadoop client? Normally I just have to use "kinit" to get
> a kerberos ticket and then I am authenticated to invoke on HDFS. How does
> HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created a
> > new branch "has-project" in Kerby, HAS is under "has" folder. Please
> > look at https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication. HAS
> > aims to build a standalone authentication service for the big data
> > ecosystem that simplifies the support of Kerberos and allows to use
> > more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/support
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it
> > more solid and stronger. And if HAS can be merged to Apache Kerby,
> > community will help HAS grow faster and users can more easily using
> > this solution in their own production. We have two suggestions about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under this
> > module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to provide
> > us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Thanks Colm.
> It sounds like a really interesting project.
I'm glad to here that.
> Have you got any feedback from the Hadoop project about it?
We haven't proposed this solution in the hadoop community.
> I'm finding it hard to understand exactly how it works though based on the README. Could you describe how it works from a really basic point of view for say a simple Hadoop client? Normally I just have to use "kinit" to get a kerberos ticket and then I am authenticated to invoke on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
Following are the steps of user accessing HDFS service, taking the cmd "hadoop fs -ls /" as an example:
1. user runs the command "hadoop fs -ls /"
2. Hadoop client will call the "HasLoginModule", https://github.com/apache/directory-kerby/blob/has-project/has/has-client/src/main/java/org/apache/hadoop/has/client/HasLoginModule.java
3. "HasLoginModule" will call the "HasClient", https://github.com/apache/directory-kerby/blob/438904f7e557a085c8c336efd2d2be304291d246/has/has-client/src/main/java/org/apache/hadoop/has/client/HasLoginModule.java#L237
4. "HasClient" will get the plugin type from config, then choose the right client plugin, the client plugin will collect and add some user info to "AuthToken", the following is the client plugin interface:
// Get the login module type ID, used to distinguish this module from others.
// Should correspond to the server side module.
String getLoginType()
// Perform all the client side login logics, the results wrapped in an AuthToken,
// will be validated by HAS server.
AuthToken login(Conf loginConf) throws HasLoginException
5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
6. After HAS server receives the message, it will call the server plugin, server plugin will verify the user info in AuthToken, the following is the server plugin interface:
// Get the login module type ID, used to distinguish this module from others.
// Should correspond to the client side module.
String getLoginType()
// Perform all the server side authentication logics, the results wrapped in an "AuthToken",
// will be used to exchange a Kerberos ticket.
AuthToken authenticate(AuthToken userToken) throws HasAuthenException
7. If the user info is verified in existing user authentication system, server plugin will return the verified "AuthToken" to Kerby KDC
8. Kerby KDC will issue the TGT ticket using the TokenPreauth, then send the TGT to HasClient through HTTPS
9. Now user login successful, could continue the others steps, such as: getting SGT ticket.
We replace the step through "kinit" to get Kerberos Ticket. There are two important benefits:
1. The user's principal may not be in the backend, security admins won't have to migrate and sync up their user accounts to Kerberos back and forth.
2. Multiple users could run the job at the same time and in the same machine, through collecting user info from environment variables in step4.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Monday, November 27, 2017 6:54 PM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
It sounds like a really interesting project. Have you got any feedback from the Hadoop project about it?
I'm finding it hard to understand exactly how it works though based on the README. Could you describe how it works from a really basic point of view for say a simple Hadoop client? Normally I just have to use "kinit" to get a kerberos ticket and then I am authenticated to invoke on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
Colm.
On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com> wrote:
> Hi all,
>
> I would like to post a proposal about merging a new project HAS
> (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> and Alibaba, it is a solution to support the authentication of open
> source big data ecosystem in cloud computing platforms. I've created a
> new branch "has-project" in Kerby, HAS is under "has" folder. Please
> look at https://github.com/apache/directory-kerby/tree/has-project/has
> for details.
>
> Background and motivation:
> At present, the open source big data ecosystems (Hadoop/Spark) only
> has the built-in Kerberos support on the security authentication. HAS
> aims to build a standalone authentication service for the big data
> ecosystem that simplifies the support of Kerberos and allows to use
> more authentication methods.
>
> Targets users:
> HAS supports various authentication mechanisms other than just
> Kerberos, and it provides a new authentication mechanism can be easy
> customized and plugin with existing user authentication and
> authorization system, and security admins won't have to migrate and
> sync up their user accounts to Kerberos back and forth.
>
> Architecture & Design:
> HAS provides a new authentication mechanism ("Kerberos-based token
> authentication"), depending on the "TokenPreauth" provided by Apache Kerby.
> Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/README.md for details.
>
> Features:
> 1. Provides new authentication mechanism plugin APIs to customize and
> plugin with existing user authentication and authorization system.
> Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/README.md for details.
> 2. Provides lots of REST APIs and facility tools to simplify the
> support of Kerberos. Kerberos is essentially a protocol, or secure
> channel, doesn't have to be that complex to users. Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/doc/rest-api.md for details.
> 3. Provides MySQL backend for High Availability. Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/doc/mysql-backend.md for details.
> 4. New authentication mechanism now supports most of the components
> of open source big data ecosystem with little or no changes to
> components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> look at
> https://github.com/apache/directory-kerby/tree/has-project/has/support
> s
> for details.
>
> Practice
> This solution has been deployed in Alibaba Cloud E-MapReduce production.
>
> Why to merge?
> HAS provides a complete Hadoop/Spark authentication framework and
> solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it
> more solid and stronger. And if HAS can be merged to Apache Kerby,
> community will help HAS grow faster and users can more easily using
> this solution in their own production. We have two suggestions about how to merge:
> - Option1:
> Create a standalone module "kerby-has", putting HAS project under this
> module.
> - Option2:
> Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
>
> Contributors:
> Jiajia, Li (Intel)
> Lin, Zeng (Intel)
> Zhiqiang, Zhang (Intel)
> Kai, Zheng (Intel)
> Wei, Wu (Alibaba)
> Jun, Song (Alibaba)
> Long, Cao (Alibaba)
> Zhenyuan, Wei (Alibaba)
>
> Your review efforts are truly appreciated, please feel free to provide
> us your feedback.
>
> Regards,
> Jiajia
>
>
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Thanks Colm.
> It sounds like a really interesting project.
I'm glad to here that.
> Have you got any feedback from the Hadoop project about it?
We haven't proposed this solution in the hadoop community.
> I'm finding it hard to understand exactly how it works though based on the README. Could you describe how it works from a really basic point of view for say a simple Hadoop client? Normally I just have to use "kinit" to get a kerberos ticket and then I am authenticated to invoke on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
Following are the steps of user accessing HDFS service, taking the cmd "hadoop fs -ls /" as an example:
1. user runs the command "hadoop fs -ls /"
2. Hadoop client will call the "HasLoginModule", https://github.com/apache/directory-kerby/blob/has-project/has/has-client/src/main/java/org/apache/hadoop/has/client/HasLoginModule.java
3. "HasLoginModule" will call the "HasClient", https://github.com/apache/directory-kerby/blob/438904f7e557a085c8c336efd2d2be304291d246/has/has-client/src/main/java/org/apache/hadoop/has/client/HasLoginModule.java#L237
4. "HasClient" will get the plugin type from config, then choose the right client plugin, the client plugin will collect and add some user info to "AuthToken", the following is the client plugin interface:
// Get the login module type ID, used to distinguish this module from others.
// Should correspond to the server side module.
String getLoginType()
// Perform all the client side login logics, the results wrapped in an AuthToken,
// will be validated by HAS server.
AuthToken login(Conf loginConf) throws HasLoginException
5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
6. After HAS server receives the message, it will call the server plugin, server plugin will verify the user info in AuthToken, the following is the server plugin interface:
// Get the login module type ID, used to distinguish this module from others.
// Should correspond to the client side module.
String getLoginType()
// Perform all the server side authentication logics, the results wrapped in an "AuthToken",
// will be used to exchange a Kerberos ticket.
AuthToken authenticate(AuthToken userToken) throws HasAuthenException
7. If the user info is verified in existing user authentication system, server plugin will return the verified "AuthToken" to Kerby KDC
8. Kerby KDC will issue the TGT ticket using the TokenPreauth, then send the TGT to HasClient through HTTPS
9. Now user login successful, could continue the others steps, such as: getting SGT ticket.
We replace the step through "kinit" to get Kerberos Ticket. There are two important benefits:
1. The user's principal may not be in the backend, security admins won't have to migrate and sync up their user accounts to Kerberos back and forth.
2. Multiple users could run the job at the same time and in the same machine, through collecting user info from environment variables in step4.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Monday, November 27, 2017 6:54 PM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
It sounds like a really interesting project. Have you got any feedback from the Hadoop project about it?
I'm finding it hard to understand exactly how it works though based on the README. Could you describe how it works from a really basic point of view for say a simple Hadoop client? Normally I just have to use "kinit" to get a kerberos ticket and then I am authenticated to invoke on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
Colm.
On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com> wrote:
> Hi all,
>
> I would like to post a proposal about merging a new project HAS
> (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> and Alibaba, it is a solution to support the authentication of open
> source big data ecosystem in cloud computing platforms. I've created a
> new branch "has-project" in Kerby, HAS is under "has" folder. Please
> look at https://github.com/apache/directory-kerby/tree/has-project/has
> for details.
>
> Background and motivation:
> At present, the open source big data ecosystems (Hadoop/Spark) only
> has the built-in Kerberos support on the security authentication. HAS
> aims to build a standalone authentication service for the big data
> ecosystem that simplifies the support of Kerberos and allows to use
> more authentication methods.
>
> Targets users:
> HAS supports various authentication mechanisms other than just
> Kerberos, and it provides a new authentication mechanism can be easy
> customized and plugin with existing user authentication and
> authorization system, and security admins won't have to migrate and
> sync up their user accounts to Kerberos back and forth.
>
> Architecture & Design:
> HAS provides a new authentication mechanism ("Kerberos-based token
> authentication"), depending on the "TokenPreauth" provided by Apache Kerby.
> Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/README.md for details.
>
> Features:
> 1. Provides new authentication mechanism plugin APIs to customize and
> plugin with existing user authentication and authorization system.
> Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/README.md for details.
> 2. Provides lots of REST APIs and facility tools to simplify the
> support of Kerberos. Kerberos is essentially a protocol, or secure
> channel, doesn't have to be that complex to users. Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/doc/rest-api.md for details.
> 3. Provides MySQL backend for High Availability. Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/doc/mysql-backend.md for details.
> 4. New authentication mechanism now supports most of the components
> of open source big data ecosystem with little or no changes to
> components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> look at
> https://github.com/apache/directory-kerby/tree/has-project/has/support
> s
> for details.
>
> Practice
> This solution has been deployed in Alibaba Cloud E-MapReduce production.
>
> Why to merge?
> HAS provides a complete Hadoop/Spark authentication framework and
> solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it
> more solid and stronger. And if HAS can be merged to Apache Kerby,
> community will help HAS grow faster and users can more easily using
> this solution in their own production. We have two suggestions about how to merge:
> - Option1:
> Create a standalone module "kerby-has", putting HAS project under this
> module.
> - Option2:
> Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
>
> Contributors:
> Jiajia, Li (Intel)
> Lin, Zeng (Intel)
> Zhiqiang, Zhang (Intel)
> Kai, Zheng (Intel)
> Wei, Wu (Alibaba)
> Jun, Song (Alibaba)
> Long, Cao (Alibaba)
> Zhenyuan, Wei (Alibaba)
>
> Your review efforts are truly appreciated, please feel free to provide
> us your feedback.
>
> Regards,
> Jiajia
>
>
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jiajia,
It sounds like a really interesting project. Have you got any feedback from
the Hadoop project about it?
I'm finding it hard to understand exactly how it works though based on the
README. Could you describe how it works from a really basic point of view
for say a simple Hadoop client? Normally I just have to use "kinit" to get
a kerberos ticket and then I am authenticated to invoke on HDFS. How does
HAS work differently? Where does the token pre-auth stuff fit in?
Colm.
On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com> wrote:
> Hi all,
>
> I would like to post a proposal about merging a new project HAS (Hadoop
> Authentication Service) to Apache Kerby. HAS is led by Intel and Alibaba,
> it is a solution to support the authentication of open source big data
> ecosystem in cloud computing platforms. I've created a new branch
> "has-project" in Kerby, HAS is under "has" folder. Please look at
> https://github.com/apache/directory-kerby/tree/has-project/has for
> details.
>
> Background and motivation:
> At present, the open source big data ecosystems (Hadoop/Spark) only has
> the built-in Kerberos support on the security authentication. HAS aims to
> build a standalone authentication service for the big data ecosystem that
> simplifies the support of Kerberos and allows to use more authentication
> methods.
>
> Targets users:
> HAS supports various authentication mechanisms other than just Kerberos,
> and it provides a new authentication mechanism can be easy customized and
> plugin with existing user authentication and authorization system, and
> security admins won't have to migrate and sync up their user accounts to
> Kerberos back and forth.
>
> Architecture & Design:
> HAS provides a new authentication mechanism ("Kerberos-based token
> authentication"), depending on the "TokenPreauth" provided by Apache Kerby.
> Please look at https://github.com/apache/directory-kerby/blob/has-project/
> has/README.md for details.
>
> Features:
> 1. Provides new authentication mechanism plugin APIs to customize and
> plugin with existing user authentication and authorization system. Please
> look at https://github.com/apache/directory-kerby/blob/has-project/
> has/README.md for details.
> 2. Provides lots of REST APIs and facility tools to simplify the
> support of Kerberos. Kerberos is essentially a protocol, or secure channel,
> doesn't have to be that complex to users. Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/doc/rest-api.md for details.
> 3. Provides MySQL backend for High Availability. Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/doc/mysql-backend.md for details.
> 4. New authentication mechanism now supports most of the components
> of open source big data ecosystem with little or no changes to components,
> including HDFS, HBase, Zookeeper, Hive, Spark.... Please look at
> https://github.com/apache/directory-kerby/tree/has-project/has/supports
> for details.
>
> Practice
> This solution has been deployed in Alibaba Cloud E-MapReduce production.
>
> Why to merge?
> HAS provides a complete Hadoop/Spark authentication framework and solution
> based on Kerberos, HAS can help to upgrade Kerby KDC, make it more solid
> and stronger. And if HAS can be merged to Apache Kerby, community will help
> HAS grow faster and users can more easily using this solution in their own
> production. We have two suggestions about how to merge:
> - Option1:
> Create a standalone module "kerby-has", putting HAS project under this
> module.
> - Option2:
> Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
>
> Contributors:
> Jiajia, Li (Intel)
> Lin, Zeng (Intel)
> Zhiqiang, Zhang (Intel)
> Kai, Zheng (Intel)
> Wei, Wu (Alibaba)
> Jun, Song (Alibaba)
> Long, Cao (Alibaba)
> Zhenyuan, Wei (Alibaba)
>
> Your review efforts are truly appreciated, please feel free to provide us
> your feedback.
>
> Regards,
> Jiajia
>
>
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
On Mon, Nov 27, 2017 at 6:19 AM, Li, Jiajia <ji...@intel.com> wrote:
>
>
> We will fix these dependencies soon. One question: I found Apache Hadoop
> also using the Jersey, could we using it through adding the license file?
>
I think Jersey is OK as it is CDDL:
https://www.apache.org/legal/resolved.html
"Software under the following licenses may be included in binary form
within an Apache product if the inclusion is appropriately labeled (see
below):
- Common Development and Distribution Licenses: CDDL 1.0
<https://opensource.org/licenses/CDDL-1.0> and CDDL 1.1
<https://spdx.org/licenses/CDDL-1.1.html>"
Colm.
>
>
> > Plugins, especially "RAM":
> > * What does "RAM" mean?
> > * The RAM plugin is not included but tests and default config seem to
> require it. Classes org.apache.hadoop.has.plugins.client.aliyun.AliyunHasClientPlugin
> and org.apache.hadoop.has.plugins.server.aliyun.AliyunHasServerPlugin are
> not available. Will those also be contributed?
> > * There is no other (default) implementation of HasClientPlugin. Is the
> project still already usable? Or is it only a framework and more
> development effort is requried to implement the plugins?
>
> "RAM" is an example plugin type, It's the name of existing user
> authentication system. I think it's universal enough, so I removed all the
> code associated with this plugin. This project is a framework, users should
> implement their own plugins(including client and server). But we will
> provide the default implementation of HasClientPlugin in the future work.
> Do you have any suggestions for default plugin? LDAP or others?
>
>
> > Hadoop or Kerby/Directory:
> > * The project name includes Hadoop, the Maven groupId is
> org.apache.hadoop, Java package names are org.apache.hadoop.has. Was it
> planned to contribute this to Hadoop? Or does it make more sense to
> contribute it to Hadoop directly?
>
> No, we won't contribute it to Hadoop. We will change the Maven groupId to
> "org.apache.kerby".
>
>
> > * On the other hand it seems to be also useful otherwise, like to
> configure Kerby KDC via REST and to be able to plugin other authentication
> providers, am I right? Then it totally makes sense to include it into
> Kerby. But in that case I'd suggest to change the names.
>
> Yes, you are right, HAS provides lots of REST APIs to config Kerby KDC,
> and the new authentication mechanism is able to plugin the existing user
> authentication system. I thinks it's ok to change the name of "HAS" if we
> merge it to replace kerby-kdc module to upgrade the Kerby KDC, and do you
> have a suggest name?
> On the other hand, "HAS" also provides the supports for Hadoop ecosystem,
> it's a complete framework for Hadoop Ecosystem, so I think "HAS" is a good
> choice if we want to create a standalone module for it. Please correct me
> if I am wrong.
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Stefan Seelmann [mailto:mail@stefan-seelmann.de]
> Sent: Sunday, November 26, 2017 12:22 AM
> To: Apache Directory Developers List <de...@directory.apache.org>;
> kerby@directory.apache.org
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> I browsed the code and documentation, here some notes and questions:
>
> Was the HAS developed as open source project in public? In readme I see
> some links to github.com/intel-bigdata/has but that only gives 404.
>
> Are all the contributors already ASF committers or have an ICLA on file?
> Otherwise I'm afraid IP clearance is required.
>
> Dependencies:
> * MySQL JDBC driver is GPLv2 which is not compatible with Apache.
> (alternative: Drizzle JDBC)
> * Some dependencies (Jersey, Glassfish) are CDDL licensed which is not
> compatible with Apache. (alternatives: CXF, Geronimo)
> * For some dependencies I cannot find a license:
> com.aliyun:aliyun-java-sdk-core and com.aliyun:aliyun-java-sdk-ram
>
> Plugins, especially "RAM":
> * What does "RAM" mean?
> * The RAM plugin is not included but tests and default config seem to
> require it. Classes org.apache.hadoop.has.plugins.client.aliyun.AliyunHasClientPlugin
> and org.apache.hadoop.has.plugins.server.aliyun.AliyunHasServerPlugin are
> not available. Will those also be contributed?
> * There is no other (default) implementation of HasClientPlugin. Is the
> project still already usable? Or is it only a framework and more
> development effort is requried to implement the plugins?
>
> Hadoop or Kerby/Directory:
> * The project name includes Hadoop, the Maven groupId is
> org.apache.hadoop, Java package names are org.apache.hadoop.has. Was it
> planned to contribute this to Hadoop? Or does it make more sense to
> contribute it to Hadoop directly?
> * On the other hand it seems to be also useful otherwise, like to
> configure Kerby KDC via REST and to be able to plugin other authentication
> providers, am I right? Then it totally makes sense to include it into
> Kerby. But in that case I'd suggest to change the names.
>
> Kind Regards,
> Stefan
>
>
>
> On 11/24/2017 04:30 AM, Li, Jiajia wrote:
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS (Hadoop
> Authentication Service) to Apache Kerby. HAS is led by Intel and Alibaba,
> it is a solution to support the authentication of open source big data
> ecosystem in cloud computing platforms. I've created a new branch
> "has-project" in Kerby, HAS is under "has" folder. Please look at
> https://github.com/apache/directory-kerby/tree/has-project/has for
> details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only has
> the built-in Kerberos support on the security authentication. HAS aims to
> build a standalone authentication service for the big data ecosystem that
> simplifies the support of Kerberos and allows to use more authentication
> methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just Kerberos,
> and it provides a new authentication mechanism can be easy customized and
> plugin with existing user authentication and authorization system, and
> security admins won't have to migrate and sync up their user accounts to
> Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> authentication"), depending on the "TokenPreauth" provided by Apache Kerby.
> Please look at https://github.com/apache/directory-kerby/blob/has-
> project/has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and plugin with existing user authentication and authorization system.
> Please look at https://github.com/apache/directory-kerby/blob/has-
> project/has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> support of Kerberos. Kerberos is essentially a protocol, or secure channel,
> doesn't have to be that complex to users. Please look at
> https://github.com/apache/directory-kerby/blob/has-
> project/has/doc/rest-api.md for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> https://github.com/apache/directory-kerby/blob/has-
> project/has/doc/mysql-backend.md for details.
> > 4. New authentication mechanism now supports most of the components
> of open source big data ecosystem with little or no changes to components,
> including HDFS, HBase, Zookeeper, Hive, Spark.... Please look at
> https://github.com/apache/directory-kerby/tree/has-project/has/supports
> for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it more
> solid and stronger. And if HAS can be merged to Apache Kerby, community
> will help HAS grow faster and users can more easily using this solution in
> their own production. We have two suggestions about how to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under this
> module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to provide
> us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
On Mon, Nov 27, 2017 at 6:19 AM, Li, Jiajia <ji...@intel.com> wrote:
>
>
> We will fix these dependencies soon. One question: I found Apache Hadoop
> also using the Jersey, could we using it through adding the license file?
>
I think Jersey is OK as it is CDDL:
https://www.apache.org/legal/resolved.html
"Software under the following licenses may be included in binary form
within an Apache product if the inclusion is appropriately labeled (see
below):
- Common Development and Distribution Licenses: CDDL 1.0
<https://opensource.org/licenses/CDDL-1.0> and CDDL 1.1
<https://spdx.org/licenses/CDDL-1.1.html>"
Colm.
>
>
> > Plugins, especially "RAM":
> > * What does "RAM" mean?
> > * The RAM plugin is not included but tests and default config seem to
> require it. Classes org.apache.hadoop.has.plugins.client.aliyun.AliyunHasClientPlugin
> and org.apache.hadoop.has.plugins.server.aliyun.AliyunHasServerPlugin are
> not available. Will those also be contributed?
> > * There is no other (default) implementation of HasClientPlugin. Is the
> project still already usable? Or is it only a framework and more
> development effort is requried to implement the plugins?
>
> "RAM" is an example plugin type, It's the name of existing user
> authentication system. I think it's universal enough, so I removed all the
> code associated with this plugin. This project is a framework, users should
> implement their own plugins(including client and server). But we will
> provide the default implementation of HasClientPlugin in the future work.
> Do you have any suggestions for default plugin? LDAP or others?
>
>
> > Hadoop or Kerby/Directory:
> > * The project name includes Hadoop, the Maven groupId is
> org.apache.hadoop, Java package names are org.apache.hadoop.has. Was it
> planned to contribute this to Hadoop? Or does it make more sense to
> contribute it to Hadoop directly?
>
> No, we won't contribute it to Hadoop. We will change the Maven groupId to
> "org.apache.kerby".
>
>
> > * On the other hand it seems to be also useful otherwise, like to
> configure Kerby KDC via REST and to be able to plugin other authentication
> providers, am I right? Then it totally makes sense to include it into
> Kerby. But in that case I'd suggest to change the names.
>
> Yes, you are right, HAS provides lots of REST APIs to config Kerby KDC,
> and the new authentication mechanism is able to plugin the existing user
> authentication system. I thinks it's ok to change the name of "HAS" if we
> merge it to replace kerby-kdc module to upgrade the Kerby KDC, and do you
> have a suggest name?
> On the other hand, "HAS" also provides the supports for Hadoop ecosystem,
> it's a complete framework for Hadoop Ecosystem, so I think "HAS" is a good
> choice if we want to create a standalone module for it. Please correct me
> if I am wrong.
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Stefan Seelmann [mailto:mail@stefan-seelmann.de]
> Sent: Sunday, November 26, 2017 12:22 AM
> To: Apache Directory Developers List <de...@directory.apache.org>;
> kerby@directory.apache.org
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> I browsed the code and documentation, here some notes and questions:
>
> Was the HAS developed as open source project in public? In readme I see
> some links to github.com/intel-bigdata/has but that only gives 404.
>
> Are all the contributors already ASF committers or have an ICLA on file?
> Otherwise I'm afraid IP clearance is required.
>
> Dependencies:
> * MySQL JDBC driver is GPLv2 which is not compatible with Apache.
> (alternative: Drizzle JDBC)
> * Some dependencies (Jersey, Glassfish) are CDDL licensed which is not
> compatible with Apache. (alternatives: CXF, Geronimo)
> * For some dependencies I cannot find a license:
> com.aliyun:aliyun-java-sdk-core and com.aliyun:aliyun-java-sdk-ram
>
> Plugins, especially "RAM":
> * What does "RAM" mean?
> * The RAM plugin is not included but tests and default config seem to
> require it. Classes org.apache.hadoop.has.plugins.client.aliyun.AliyunHasClientPlugin
> and org.apache.hadoop.has.plugins.server.aliyun.AliyunHasServerPlugin are
> not available. Will those also be contributed?
> * There is no other (default) implementation of HasClientPlugin. Is the
> project still already usable? Or is it only a framework and more
> development effort is requried to implement the plugins?
>
> Hadoop or Kerby/Directory:
> * The project name includes Hadoop, the Maven groupId is
> org.apache.hadoop, Java package names are org.apache.hadoop.has. Was it
> planned to contribute this to Hadoop? Or does it make more sense to
> contribute it to Hadoop directly?
> * On the other hand it seems to be also useful otherwise, like to
> configure Kerby KDC via REST and to be able to plugin other authentication
> providers, am I right? Then it totally makes sense to include it into
> Kerby. But in that case I'd suggest to change the names.
>
> Kind Regards,
> Stefan
>
>
>
> On 11/24/2017 04:30 AM, Li, Jiajia wrote:
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS (Hadoop
> Authentication Service) to Apache Kerby. HAS is led by Intel and Alibaba,
> it is a solution to support the authentication of open source big data
> ecosystem in cloud computing platforms. I've created a new branch
> "has-project" in Kerby, HAS is under "has" folder. Please look at
> https://github.com/apache/directory-kerby/tree/has-project/has for
> details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only has
> the built-in Kerberos support on the security authentication. HAS aims to
> build a standalone authentication service for the big data ecosystem that
> simplifies the support of Kerberos and allows to use more authentication
> methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just Kerberos,
> and it provides a new authentication mechanism can be easy customized and
> plugin with existing user authentication and authorization system, and
> security admins won't have to migrate and sync up their user accounts to
> Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> authentication"), depending on the "TokenPreauth" provided by Apache Kerby.
> Please look at https://github.com/apache/directory-kerby/blob/has-
> project/has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and plugin with existing user authentication and authorization system.
> Please look at https://github.com/apache/directory-kerby/blob/has-
> project/has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> support of Kerberos. Kerberos is essentially a protocol, or secure channel,
> doesn't have to be that complex to users. Please look at
> https://github.com/apache/directory-kerby/blob/has-
> project/has/doc/rest-api.md for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> https://github.com/apache/directory-kerby/blob/has-
> project/has/doc/mysql-backend.md for details.
> > 4. New authentication mechanism now supports most of the components
> of open source big data ecosystem with little or no changes to components,
> including HDFS, HBase, Zookeeper, Hive, Spark.... Please look at
> https://github.com/apache/directory-kerby/tree/has-project/has/supports
> for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it more
> solid and stronger. And if HAS can be merged to Apache Kerby, community
> will help HAS grow faster and users can more easily using this solution in
> their own production. We have two suggestions about how to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under this
> module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to provide
> us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Thanks Stefan.
> Was the HAS developed as open source project in public?
No, HAS is a private repo under https://github.com/Intel-bigdata.
> In readme I see some links to github.com/intel-bigdata/has but that only gives 404.
Sorry for my mistake, I've fixed the links.
Dependencies:
> * MySQL JDBC driver is GPLv2 which is not compatible with Apache.
(alternative: Drizzle JDBC)
> * Some dependencies (Jersey, Glassfish) are CDDL licensed which is not compatible with Apache. (alternatives: CXF, Geronimo)
> * For some dependencies I cannot find a license:
> com.aliyun:aliyun-java-sdk-core and com.aliyun:aliyun-java-sdk-ram
We will fix these dependencies soon. One question: I found Apache Hadoop also using the Jersey, could we using it through adding the license file?
> Plugins, especially "RAM":
> * What does "RAM" mean?
> * The RAM plugin is not included but tests and default config seem to require it. Classes org.apache.hadoop.has.plugins.client.aliyun.AliyunHasClientPlugin and org.apache.hadoop.has.plugins.server.aliyun.AliyunHasServerPlugin are not available. Will those also be contributed?
> * There is no other (default) implementation of HasClientPlugin. Is the project still already usable? Or is it only a framework and more development effort is requried to implement the plugins?
"RAM" is an example plugin type, It's the name of existing user authentication system. I think it's universal enough, so I removed all the code associated with this plugin. This project is a framework, users should implement their own plugins(including client and server). But we will provide the default implementation of HasClientPlugin in the future work. Do you have any suggestions for default plugin? LDAP or others?
> Hadoop or Kerby/Directory:
> * The project name includes Hadoop, the Maven groupId is org.apache.hadoop, Java package names are org.apache.hadoop.has. Was it planned to contribute this to Hadoop? Or does it make more sense to contribute it to Hadoop directly?
No, we won't contribute it to Hadoop. We will change the Maven groupId to "org.apache.kerby".
> * On the other hand it seems to be also useful otherwise, like to configure Kerby KDC via REST and to be able to plugin other authentication providers, am I right? Then it totally makes sense to include it into Kerby. But in that case I'd suggest to change the names.
Yes, you are right, HAS provides lots of REST APIs to config Kerby KDC, and the new authentication mechanism is able to plugin the existing user authentication system. I thinks it's ok to change the name of "HAS" if we merge it to replace kerby-kdc module to upgrade the Kerby KDC, and do you have a suggest name?
On the other hand, "HAS" also provides the supports for Hadoop ecosystem, it's a complete framework for Hadoop Ecosystem, so I think "HAS" is a good choice if we want to create a standalone module for it. Please correct me if I am wrong.
Thanks,
Jiajia
-----Original Message-----
From: Stefan Seelmann [mailto:mail@stefan-seelmann.de]
Sent: Sunday, November 26, 2017 12:22 AM
To: Apache Directory Developers List <de...@directory.apache.org>; kerby@directory.apache.org
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
I browsed the code and documentation, here some notes and questions:
Was the HAS developed as open source project in public? In readme I see some links to github.com/intel-bigdata/has but that only gives 404.
Are all the contributors already ASF committers or have an ICLA on file?
Otherwise I'm afraid IP clearance is required.
Dependencies:
* MySQL JDBC driver is GPLv2 which is not compatible with Apache.
(alternative: Drizzle JDBC)
* Some dependencies (Jersey, Glassfish) are CDDL licensed which is not compatible with Apache. (alternatives: CXF, Geronimo)
* For some dependencies I cannot find a license:
com.aliyun:aliyun-java-sdk-core and com.aliyun:aliyun-java-sdk-ram
Plugins, especially "RAM":
* What does "RAM" mean?
* The RAM plugin is not included but tests and default config seem to require it. Classes org.apache.hadoop.has.plugins.client.aliyun.AliyunHasClientPlugin and org.apache.hadoop.has.plugins.server.aliyun.AliyunHasServerPlugin are not available. Will those also be contributed?
* There is no other (default) implementation of HasClientPlugin. Is the project still already usable? Or is it only a framework and more development effort is requried to implement the plugins?
Hadoop or Kerby/Directory:
* The project name includes Hadoop, the Maven groupId is org.apache.hadoop, Java package names are org.apache.hadoop.has. Was it planned to contribute this to Hadoop? Or does it make more sense to contribute it to Hadoop directly?
* On the other hand it seems to be also useful otherwise, like to configure Kerby KDC via REST and to be able to plugin other authentication providers, am I right? Then it totally makes sense to include it into Kerby. But in that case I'd suggest to change the names.
Kind Regards,
Stefan
On 11/24/2017 04:30 AM, Li, Jiajia wrote:
> Hi all,
>
> I would like to post a proposal about merging a new project HAS (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel and Alibaba, it is a solution to support the authentication of open source big data ecosystem in cloud computing platforms. I've created a new branch "has-project" in Kerby, HAS is under "has" folder. Please look at https://github.com/apache/directory-kerby/tree/has-project/has for details.
>
> Background and motivation:
> At present, the open source big data ecosystems (Hadoop/Spark) only has the built-in Kerberos support on the security authentication. HAS aims to build a standalone authentication service for the big data ecosystem that simplifies the support of Kerberos and allows to use more authentication methods.
>
> Targets users:
> HAS supports various authentication mechanisms other than just Kerberos, and it provides a new authentication mechanism can be easy customized and plugin with existing user authentication and authorization system, and security admins won't have to migrate and sync up their user accounts to Kerberos back and forth.
>
> Architecture & Design:
> HAS provides a new authentication mechanism ("Kerberos-based token authentication"), depending on the "TokenPreauth" provided by Apache Kerby. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
>
> Features:
> 1. Provides new authentication mechanism plugin APIs to customize and plugin with existing user authentication and authorization system. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
> 2. Provides lots of REST APIs and facility tools to simplify the support of Kerberos. Kerberos is essentially a protocol, or secure channel, doesn't have to be that complex to users. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/rest-api.md for details.
> 3. Provides MySQL backend for High Availability. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/mysql-backend.md for details.
> 4. New authentication mechanism now supports most of the components of open source big data ecosystem with little or no changes to components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please look at https://github.com/apache/directory-kerby/tree/has-project/has/supports for details.
>
> Practice
> This solution has been deployed in Alibaba Cloud E-MapReduce production.
>
> Why to merge?
> HAS provides a complete Hadoop/Spark authentication framework and solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it more solid and stronger. And if HAS can be merged to Apache Kerby, community will help HAS grow faster and users can more easily using this solution in their own production. We have two suggestions about how to merge:
> - Option1:
> Create a standalone module "kerby-has", putting HAS project under this module.
> - Option2:
> Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
>
> Contributors:
> Jiajia, Li (Intel)
> Lin, Zeng (Intel)
> Zhiqiang, Zhang (Intel)
> Kai, Zheng (Intel)
> Wei, Wu (Alibaba)
> Jun, Song (Alibaba)
> Long, Cao (Alibaba)
> Zhenyuan, Wei (Alibaba)
>
> Your review efforts are truly appreciated, please feel free to provide us your feedback.
>
> Regards,
> Jiajia
>
>
>
>
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Thanks Stefan.
> Was the HAS developed as open source project in public?
No, HAS is a private repo under https://github.com/Intel-bigdata.
> In readme I see some links to github.com/intel-bigdata/has but that only gives 404.
Sorry for my mistake, I've fixed the links.
Dependencies:
> * MySQL JDBC driver is GPLv2 which is not compatible with Apache.
(alternative: Drizzle JDBC)
> * Some dependencies (Jersey, Glassfish) are CDDL licensed which is not compatible with Apache. (alternatives: CXF, Geronimo)
> * For some dependencies I cannot find a license:
> com.aliyun:aliyun-java-sdk-core and com.aliyun:aliyun-java-sdk-ram
We will fix these dependencies soon. One question: I found Apache Hadoop also using the Jersey, could we using it through adding the license file?
> Plugins, especially "RAM":
> * What does "RAM" mean?
> * The RAM plugin is not included but tests and default config seem to require it. Classes org.apache.hadoop.has.plugins.client.aliyun.AliyunHasClientPlugin and org.apache.hadoop.has.plugins.server.aliyun.AliyunHasServerPlugin are not available. Will those also be contributed?
> * There is no other (default) implementation of HasClientPlugin. Is the project still already usable? Or is it only a framework and more development effort is requried to implement the plugins?
"RAM" is an example plugin type, It's the name of existing user authentication system. I think it's universal enough, so I removed all the code associated with this plugin. This project is a framework, users should implement their own plugins(including client and server). But we will provide the default implementation of HasClientPlugin in the future work. Do you have any suggestions for default plugin? LDAP or others?
> Hadoop or Kerby/Directory:
> * The project name includes Hadoop, the Maven groupId is org.apache.hadoop, Java package names are org.apache.hadoop.has. Was it planned to contribute this to Hadoop? Or does it make more sense to contribute it to Hadoop directly?
No, we won't contribute it to Hadoop. We will change the Maven groupId to "org.apache.kerby".
> * On the other hand it seems to be also useful otherwise, like to configure Kerby KDC via REST and to be able to plugin other authentication providers, am I right? Then it totally makes sense to include it into Kerby. But in that case I'd suggest to change the names.
Yes, you are right, HAS provides lots of REST APIs to config Kerby KDC, and the new authentication mechanism is able to plugin the existing user authentication system. I thinks it's ok to change the name of "HAS" if we merge it to replace kerby-kdc module to upgrade the Kerby KDC, and do you have a suggest name?
On the other hand, "HAS" also provides the supports for Hadoop ecosystem, it's a complete framework for Hadoop Ecosystem, so I think "HAS" is a good choice if we want to create a standalone module for it. Please correct me if I am wrong.
Thanks,
Jiajia
-----Original Message-----
From: Stefan Seelmann [mailto:mail@stefan-seelmann.de]
Sent: Sunday, November 26, 2017 12:22 AM
To: Apache Directory Developers List <de...@directory.apache.org>; kerby@directory.apache.org
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
I browsed the code and documentation, here some notes and questions:
Was the HAS developed as open source project in public? In readme I see some links to github.com/intel-bigdata/has but that only gives 404.
Are all the contributors already ASF committers or have an ICLA on file?
Otherwise I'm afraid IP clearance is required.
Dependencies:
* MySQL JDBC driver is GPLv2 which is not compatible with Apache.
(alternative: Drizzle JDBC)
* Some dependencies (Jersey, Glassfish) are CDDL licensed which is not compatible with Apache. (alternatives: CXF, Geronimo)
* For some dependencies I cannot find a license:
com.aliyun:aliyun-java-sdk-core and com.aliyun:aliyun-java-sdk-ram
Plugins, especially "RAM":
* What does "RAM" mean?
* The RAM plugin is not included but tests and default config seem to require it. Classes org.apache.hadoop.has.plugins.client.aliyun.AliyunHasClientPlugin and org.apache.hadoop.has.plugins.server.aliyun.AliyunHasServerPlugin are not available. Will those also be contributed?
* There is no other (default) implementation of HasClientPlugin. Is the project still already usable? Or is it only a framework and more development effort is requried to implement the plugins?
Hadoop or Kerby/Directory:
* The project name includes Hadoop, the Maven groupId is org.apache.hadoop, Java package names are org.apache.hadoop.has. Was it planned to contribute this to Hadoop? Or does it make more sense to contribute it to Hadoop directly?
* On the other hand it seems to be also useful otherwise, like to configure Kerby KDC via REST and to be able to plugin other authentication providers, am I right? Then it totally makes sense to include it into Kerby. But in that case I'd suggest to change the names.
Kind Regards,
Stefan
On 11/24/2017 04:30 AM, Li, Jiajia wrote:
> Hi all,
>
> I would like to post a proposal about merging a new project HAS (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel and Alibaba, it is a solution to support the authentication of open source big data ecosystem in cloud computing platforms. I've created a new branch "has-project" in Kerby, HAS is under "has" folder. Please look at https://github.com/apache/directory-kerby/tree/has-project/has for details.
>
> Background and motivation:
> At present, the open source big data ecosystems (Hadoop/Spark) only has the built-in Kerberos support on the security authentication. HAS aims to build a standalone authentication service for the big data ecosystem that simplifies the support of Kerberos and allows to use more authentication methods.
>
> Targets users:
> HAS supports various authentication mechanisms other than just Kerberos, and it provides a new authentication mechanism can be easy customized and plugin with existing user authentication and authorization system, and security admins won't have to migrate and sync up their user accounts to Kerberos back and forth.
>
> Architecture & Design:
> HAS provides a new authentication mechanism ("Kerberos-based token authentication"), depending on the "TokenPreauth" provided by Apache Kerby. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
>
> Features:
> 1. Provides new authentication mechanism plugin APIs to customize and plugin with existing user authentication and authorization system. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
> 2. Provides lots of REST APIs and facility tools to simplify the support of Kerberos. Kerberos is essentially a protocol, or secure channel, doesn't have to be that complex to users. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/rest-api.md for details.
> 3. Provides MySQL backend for High Availability. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/mysql-backend.md for details.
> 4. New authentication mechanism now supports most of the components of open source big data ecosystem with little or no changes to components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please look at https://github.com/apache/directory-kerby/tree/has-project/has/supports for details.
>
> Practice
> This solution has been deployed in Alibaba Cloud E-MapReduce production.
>
> Why to merge?
> HAS provides a complete Hadoop/Spark authentication framework and solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it more solid and stronger. And if HAS can be merged to Apache Kerby, community will help HAS grow faster and users can more easily using this solution in their own production. We have two suggestions about how to merge:
> - Option1:
> Create a standalone module "kerby-has", putting HAS project under this module.
> - Option2:
> Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
>
> Contributors:
> Jiajia, Li (Intel)
> Lin, Zeng (Intel)
> Zhiqiang, Zhang (Intel)
> Kai, Zheng (Intel)
> Wei, Wu (Alibaba)
> Jun, Song (Alibaba)
> Long, Cao (Alibaba)
> Zhenyuan, Wei (Alibaba)
>
> Your review efforts are truly appreciated, please feel free to provide us your feedback.
>
> Regards,
> Jiajia
>
>
>
>
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
I browsed the code and documentation, here some notes and questions:
Was the HAS developed as open source project in public? In readme I see
some links to github.com/intel-bigdata/has but that only gives 404.
Are all the contributors already ASF committers or have an ICLA on file?
Otherwise I'm afraid IP clearance is required.
Dependencies:
* MySQL JDBC driver is GPLv2 which is not compatible with Apache.
(alternative: Drizzle JDBC)
* Some dependencies (Jersey, Glassfish) are CDDL licensed which is not
compatible with Apache. (alternatives: CXF, Geronimo)
* For some dependencies I cannot find a license:
com.aliyun:aliyun-java-sdk-core and com.aliyun:aliyun-java-sdk-ram
Plugins, especially "RAM":
* What does "RAM" mean?
* The RAM plugin is not included but tests and default config seem to
require it. Classes
org.apache.hadoop.has.plugins.client.aliyun.AliyunHasClientPlugin and
org.apache.hadoop.has.plugins.server.aliyun.AliyunHasServerPlugin are
not available. Will those also be contributed?
* There is no other (default) implementation of HasClientPlugin. Is the
project still already usable? Or is it only a framework and more
development effort is requried to implement the plugins?
Hadoop or Kerby/Directory:
* The project name includes Hadoop, the Maven groupId is
org.apache.hadoop, Java package names are org.apache.hadoop.has. Was it
planned to contribute this to Hadoop? Or does it make more sense to
contribute it to Hadoop directly?
* On the other hand it seems to be also useful otherwise, like to
configure Kerby KDC via REST and to be able to plugin other
authentication providers, am I right? Then it totally makes sense to
include it into Kerby. But in that case I'd suggest to change the names.
Kind Regards,
Stefan
On 11/24/2017 04:30 AM, Li, Jiajia wrote:
> Hi all,
>
> I would like to post a proposal about merging a new project HAS (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel and Alibaba, it is a solution to support the authentication of open source big data ecosystem in cloud computing platforms. I've created a new branch "has-project" in Kerby, HAS is under "has" folder. Please look at https://github.com/apache/directory-kerby/tree/has-project/has for details.
>
> Background and motivation:
> At present, the open source big data ecosystems (Hadoop/Spark) only has the built-in Kerberos support on the security authentication. HAS aims to build a standalone authentication service for the big data ecosystem that simplifies the support of Kerberos and allows to use more authentication methods.
>
> Targets users:
> HAS supports various authentication mechanisms other than just Kerberos, and it provides a new authentication mechanism can be easy customized and plugin with existing user authentication and authorization system, and security admins won't have to migrate and sync up their user accounts to Kerberos back and forth.
>
> Architecture & Design:
> HAS provides a new authentication mechanism ("Kerberos-based token authentication"), depending on the "TokenPreauth" provided by Apache Kerby. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
>
> Features:
> 1. Provides new authentication mechanism plugin APIs to customize and plugin with existing user authentication and authorization system. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
> 2. Provides lots of REST APIs and facility tools to simplify the support of Kerberos. Kerberos is essentially a protocol, or secure channel, doesn't have to be that complex to users. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/rest-api.md for details.
> 3. Provides MySQL backend for High Availability. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/mysql-backend.md for details.
> 4. New authentication mechanism now supports most of the components of open source big data ecosystem with little or no changes to components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please look at https://github.com/apache/directory-kerby/tree/has-project/has/supports for details.
>
> Practice
> This solution has been deployed in Alibaba Cloud E-MapReduce production.
>
> Why to merge?
> HAS provides a complete Hadoop/Spark authentication framework and solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it more solid and stronger. And if HAS can be merged to Apache Kerby, community will help HAS grow faster and users can more easily using this solution in their own production. We have two suggestions about how to merge:
> - Option1:
> Create a standalone module "kerby-has", putting HAS project under this module.
> - Option2:
> Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
>
> Contributors:
> Jiajia, Li (Intel)
> Lin, Zeng (Intel)
> Zhiqiang, Zhang (Intel)
> Kai, Zheng (Intel)
> Wei, Wu (Alibaba)
> Jun, Song (Alibaba)
> Long, Cao (Alibaba)
> Zhenyuan, Wei (Alibaba)
>
> Your review efforts are truly appreciated, please feel free to provide us your feedback.
>
> Regards,
> Jiajia
>
>
>
>
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jiajia,
It sounds like a really interesting project. Have you got any feedback from
the Hadoop project about it?
I'm finding it hard to understand exactly how it works though based on the
README. Could you describe how it works from a really basic point of view
for say a simple Hadoop client? Normally I just have to use "kinit" to get
a kerberos ticket and then I am authenticated to invoke on HDFS. How does
HAS work differently? Where does the token pre-auth stuff fit in?
Colm.
On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com> wrote:
> Hi all,
>
> I would like to post a proposal about merging a new project HAS (Hadoop
> Authentication Service) to Apache Kerby. HAS is led by Intel and Alibaba,
> it is a solution to support the authentication of open source big data
> ecosystem in cloud computing platforms. I've created a new branch
> "has-project" in Kerby, HAS is under "has" folder. Please look at
> https://github.com/apache/directory-kerby/tree/has-project/has for
> details.
>
> Background and motivation:
> At present, the open source big data ecosystems (Hadoop/Spark) only has
> the built-in Kerberos support on the security authentication. HAS aims to
> build a standalone authentication service for the big data ecosystem that
> simplifies the support of Kerberos and allows to use more authentication
> methods.
>
> Targets users:
> HAS supports various authentication mechanisms other than just Kerberos,
> and it provides a new authentication mechanism can be easy customized and
> plugin with existing user authentication and authorization system, and
> security admins won't have to migrate and sync up their user accounts to
> Kerberos back and forth.
>
> Architecture & Design:
> HAS provides a new authentication mechanism ("Kerberos-based token
> authentication"), depending on the "TokenPreauth" provided by Apache Kerby.
> Please look at https://github.com/apache/directory-kerby/blob/has-project/
> has/README.md for details.
>
> Features:
> 1. Provides new authentication mechanism plugin APIs to customize and
> plugin with existing user authentication and authorization system. Please
> look at https://github.com/apache/directory-kerby/blob/has-project/
> has/README.md for details.
> 2. Provides lots of REST APIs and facility tools to simplify the
> support of Kerberos. Kerberos is essentially a protocol, or secure channel,
> doesn't have to be that complex to users. Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/doc/rest-api.md for details.
> 3. Provides MySQL backend for High Availability. Please look at
> https://github.com/apache/directory-kerby/blob/has-project/
> has/doc/mysql-backend.md for details.
> 4. New authentication mechanism now supports most of the components
> of open source big data ecosystem with little or no changes to components,
> including HDFS, HBase, Zookeeper, Hive, Spark.... Please look at
> https://github.com/apache/directory-kerby/tree/has-project/has/supports
> for details.
>
> Practice
> This solution has been deployed in Alibaba Cloud E-MapReduce production.
>
> Why to merge?
> HAS provides a complete Hadoop/Spark authentication framework and solution
> based on Kerberos, HAS can help to upgrade Kerby KDC, make it more solid
> and stronger. And if HAS can be merged to Apache Kerby, community will help
> HAS grow faster and users can more easily using this solution in their own
> production. We have two suggestions about how to merge:
> - Option1:
> Create a standalone module "kerby-has", putting HAS project under this
> module.
> - Option2:
> Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
>
> Contributors:
> Jiajia, Li (Intel)
> Lin, Zeng (Intel)
> Zhiqiang, Zhang (Intel)
> Kai, Zheng (Intel)
> Wei, Wu (Alibaba)
> Jun, Song (Alibaba)
> Long, Cao (Alibaba)
> Zhenyuan, Wei (Alibaba)
>
> Your review efforts are truly appreciated, please feel free to provide us
> your feedback.
>
> Regards,
> Jiajia
>
>
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
I browsed the code and documentation, here some notes and questions:
Was the HAS developed as open source project in public? In readme I see
some links to github.com/intel-bigdata/has but that only gives 404.
Are all the contributors already ASF committers or have an ICLA on file?
Otherwise I'm afraid IP clearance is required.
Dependencies:
* MySQL JDBC driver is GPLv2 which is not compatible with Apache.
(alternative: Drizzle JDBC)
* Some dependencies (Jersey, Glassfish) are CDDL licensed which is not
compatible with Apache. (alternatives: CXF, Geronimo)
* For some dependencies I cannot find a license:
com.aliyun:aliyun-java-sdk-core and com.aliyun:aliyun-java-sdk-ram
Plugins, especially "RAM":
* What does "RAM" mean?
* The RAM plugin is not included but tests and default config seem to
require it. Classes
org.apache.hadoop.has.plugins.client.aliyun.AliyunHasClientPlugin and
org.apache.hadoop.has.plugins.server.aliyun.AliyunHasServerPlugin are
not available. Will those also be contributed?
* There is no other (default) implementation of HasClientPlugin. Is the
project still already usable? Or is it only a framework and more
development effort is requried to implement the plugins?
Hadoop or Kerby/Directory:
* The project name includes Hadoop, the Maven groupId is
org.apache.hadoop, Java package names are org.apache.hadoop.has. Was it
planned to contribute this to Hadoop? Or does it make more sense to
contribute it to Hadoop directly?
* On the other hand it seems to be also useful otherwise, like to
configure Kerby KDC via REST and to be able to plugin other
authentication providers, am I right? Then it totally makes sense to
include it into Kerby. But in that case I'd suggest to change the names.
Kind Regards,
Stefan
On 11/24/2017 04:30 AM, Li, Jiajia wrote:
> Hi all,
>
> I would like to post a proposal about merging a new project HAS (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel and Alibaba, it is a solution to support the authentication of open source big data ecosystem in cloud computing platforms. I've created a new branch "has-project" in Kerby, HAS is under "has" folder. Please look at https://github.com/apache/directory-kerby/tree/has-project/has for details.
>
> Background and motivation:
> At present, the open source big data ecosystems (Hadoop/Spark) only has the built-in Kerberos support on the security authentication. HAS aims to build a standalone authentication service for the big data ecosystem that simplifies the support of Kerberos and allows to use more authentication methods.
>
> Targets users:
> HAS supports various authentication mechanisms other than just Kerberos, and it provides a new authentication mechanism can be easy customized and plugin with existing user authentication and authorization system, and security admins won't have to migrate and sync up their user accounts to Kerberos back and forth.
>
> Architecture & Design:
> HAS provides a new authentication mechanism ("Kerberos-based token authentication"), depending on the "TokenPreauth" provided by Apache Kerby. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
>
> Features:
> 1. Provides new authentication mechanism plugin APIs to customize and plugin with existing user authentication and authorization system. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
> 2. Provides lots of REST APIs and facility tools to simplify the support of Kerberos. Kerberos is essentially a protocol, or secure channel, doesn't have to be that complex to users. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/rest-api.md for details.
> 3. Provides MySQL backend for High Availability. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/mysql-backend.md for details.
> 4. New authentication mechanism now supports most of the components of open source big data ecosystem with little or no changes to components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please look at https://github.com/apache/directory-kerby/tree/has-project/has/supports for details.
>
> Practice
> This solution has been deployed in Alibaba Cloud E-MapReduce production.
>
> Why to merge?
> HAS provides a complete Hadoop/Spark authentication framework and solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it more solid and stronger. And if HAS can be merged to Apache Kerby, community will help HAS grow faster and users can more easily using this solution in their own production. We have two suggestions about how to merge:
> - Option1:
> Create a standalone module "kerby-has", putting HAS project under this module.
> - Option2:
> Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
>
> Contributors:
> Jiajia, Li (Intel)
> Lin, Zeng (Intel)
> Zhiqiang, Zhang (Intel)
> Kai, Zheng (Intel)
> Wei, Wu (Alibaba)
> Jun, Song (Alibaba)
> Long, Cao (Alibaba)
> Zhenyuan, Wei (Alibaba)
>
> Your review efforts are truly appreciated, please feel free to provide us your feedback.
>
> Regards,
> Jiajia
>
>
>
>
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Thanks Emmanuel.
> Is there some information on HAS, before it was added in a branch ?
> Typically, where does it come from (ie, the history), specs, documentation, etc ?
HAS is a private repo under https://github.com/Intel-bigdata, and I've moved all the specs and docs to https://github.com/apache/directory-kerby/tree/has-project/has/doc
> We would really need ICLA for each of those contributors who haven't already sent one, and most certain a CCLA from Intel and Alibaba.
Yes, all the contributors can provide the ICLA, I thinks Kai, Lin and me already sent one.
> Otherwise, assuming we check teh code base is 'safe ' (ie no problem with any of its dependency, and clean copyright), I would say I won't oppose to such a move.
We will take some time to check the license and copyright ASAP.
Thanks,
Jiajia
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Saturday, November 25, 2017 9:14 PM
To: kerby@directory.apache.org
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
Le 24/11/2017 à 04:30, Li, Jiajia a écrit :
> Hi all,
>
> I would like to post a proposal about merging a new project HAS (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel and Alibaba, it is a solution to support the authentication of open source big data ecosystem in cloud computing platforms. I've created a new branch "has-project" in Kerby, HAS is under "has" folder. Please look at https://github.com/apache/directory-kerby/tree/has-project/has for details.
Is there some information on HAS, before it was added in a branch ?
Typically, where does it come from (ie, the history), specs, documentation, etc ?
>
> Background and motivation:
> At present, the open source big data ecosystems (Hadoop/Spark) only has the built-in Kerberos support on the security authentication. HAS aims to build a standalone authentication service for the big data ecosystem that simplifies the support of Kerberos and allows to use more authentication methods.
>
> Targets users:
> HAS supports various authentication mechanisms other than just Kerberos, and it provides a new authentication mechanism can be easy customized and plugin with existing user authentication and authorization system, and security admins won't have to migrate and sync up their user accounts to Kerberos back and forth.
>
> Architecture & Design:
> HAS provides a new authentication mechanism ("Kerberos-based token authentication"), depending on the "TokenPreauth" provided by Apache Kerby. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
>
> Features:
> 1. Provides new authentication mechanism plugin APIs to customize and plugin with existing user authentication and authorization system. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
> 2. Provides lots of REST APIs and facility tools to simplify the support of Kerberos. Kerberos is essentially a protocol, or secure channel, doesn't have to be that complex to users. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/rest-api.md for details.
> 3. Provides MySQL backend for High Availability. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/mysql-backend.md for details.
> 4. New authentication mechanism now supports most of the components of open source big data ecosystem with little or no changes to components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please look at https://github.com/apache/directory-kerby/tree/has-project/has/supports for details.
>
> Practice
> This solution has been deployed in Alibaba Cloud E-MapReduce production.
>
> Why to merge?
> HAS provides a complete Hadoop/Spark authentication framework and solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it more solid and stronger. And if HAS can be merged to Apache Kerby, community will help HAS grow faster and users can more easily using this solution in their own production. We have two suggestions about how to merge:
> - Option1:
> Create a standalone module "kerby-has", putting HAS project under this module.
> - Option2:
> Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
>
> Contributors:
> Jiajia, Li (Intel)
> Lin, Zeng (Intel)
> Zhiqiang, Zhang (Intel)
> Kai, Zheng (Intel)
> Wei, Wu (Alibaba)
> Jun, Song (Alibaba)
> Long, Cao (Alibaba)
> Zhenyuan, Wei (Alibaba)
We would really need ICLA for each of those controbutors who haven't already sent one, and most certain a CCLA from Intel and Alibaba.
Otherwise, assuming we check teh code base is 'safe ' (ie no problem with any of its dependency, and clean copyright), I would say I won't oppose to such a move.
Keep in mind that the real key here is the maintenance of this piece of code in the long run, too...
Thanks !
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi Jiajia,
Le 24/11/2017 à 04:30, Li, Jiajia a écrit :
> Hi all,
>
> I would like to post a proposal about merging a new project HAS (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel and Alibaba, it is a solution to support the authentication of open source big data ecosystem in cloud computing platforms. I've created a new branch "has-project" in Kerby, HAS is under "has" folder. Please look at https://github.com/apache/directory-kerby/tree/has-project/has for details.
Is there some information on HAS, before it was added in a branch ?
Typically, where does it come from (ie, the history), specs,
documentation, etc ?
>
> Background and motivation:
> At present, the open source big data ecosystems (Hadoop/Spark) only has the built-in Kerberos support on the security authentication. HAS aims to build a standalone authentication service for the big data ecosystem that simplifies the support of Kerberos and allows to use more authentication methods.
>
> Targets users:
> HAS supports various authentication mechanisms other than just Kerberos, and it provides a new authentication mechanism can be easy customized and plugin with existing user authentication and authorization system, and security admins won't have to migrate and sync up their user accounts to Kerberos back and forth.
>
> Architecture & Design:
> HAS provides a new authentication mechanism ("Kerberos-based token authentication"), depending on the "TokenPreauth" provided by Apache Kerby. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
>
> Features:
> 1. Provides new authentication mechanism plugin APIs to customize and plugin with existing user authentication and authorization system. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/README.md for details.
> 2. Provides lots of REST APIs and facility tools to simplify the support of Kerberos. Kerberos is essentially a protocol, or secure channel, doesn't have to be that complex to users. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/rest-api.md for details.
> 3. Provides MySQL backend for High Availability. Please look at https://github.com/apache/directory-kerby/blob/has-project/has/doc/mysql-backend.md for details.
> 4. New authentication mechanism now supports most of the components of open source big data ecosystem with little or no changes to components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please look at https://github.com/apache/directory-kerby/tree/has-project/has/supports for details.
>
> Practice
> This solution has been deployed in Alibaba Cloud E-MapReduce production.
>
> Why to merge?
> HAS provides a complete Hadoop/Spark authentication framework and solution based on Kerberos, HAS can help to upgrade Kerby KDC, make it more solid and stronger. And if HAS can be merged to Apache Kerby, community will help HAS grow faster and users can more easily using this solution in their own production. We have two suggestions about how to merge:
> - Option1:
> Create a standalone module "kerby-has", putting HAS project under this module.
> - Option2:
> Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
>
> Contributors:
> Jiajia, Li (Intel)
> Lin, Zeng (Intel)
> Zhiqiang, Zhang (Intel)
> Kai, Zheng (Intel)
> Wei, Wu (Alibaba)
> Jun, Song (Alibaba)
> Long, Cao (Alibaba)
> Zhenyuan, Wei (Alibaba)
We would really need ICLA for each of those controbutors who haven't
already sent one, and most certain a CCLA from Intel and Alibaba.
Otherwise, assuming we check teh code base is 'safe ' (ie no problem
with any of its dependency, and clean copyright), I would say I won't
oppose to such a move.
Keep in mind that the real key here is the maintenance of this piece of
code in the long run, too...
Thanks !
--
Emmanuel Lecharny
Symas.com
directory.apache.org