You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by "Li, Jiajia" <ji...@intel.com> on 2017/12/04 06:36:30 UTC
RE: [DISCUSS] Merge HAS to Apache Kerby
Hi all,
Here with some status update, now all the contributors have provided the ICLA to secretary@apache.org and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org
Cc: kerby@directory.apache.org; Apache Directory Developers List <de...@directory.apache.org>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
+1.
Colm.
On Fri, Dec 15, 2017 at 3:52 AM, 郑锴(铁杰) <zh...@alibaba-inc.com> wrote:
> Thanks Jiajia for the update and major progress. It would be good to see
> HAS feature consolidated into Kerby KDC offering.
>
> Regards,
> Kai
>
> ------------------------------------------------------------------
> 发件人：Li, Jiajia <ji...@intel.com>
> 发送时间：2017年12月15日(星期五) 10:38
> 收件人：Apache Directory Developers List <de...@directory.apache.org>;
> coheigea@apache.org <co...@apache.org>
> 抄 送：kerby@directory.apache.org <ke...@directory.apache.org>
> 主 题：RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi all, <#m_8069559770700476617_this>
>
>
>
> Status update:
>
>
>
> 1. I've checked both Intel and Alibaba have signed the CCLA.
>
> 2. We have fixed dependency issues: mysql JDBC driver is replaced with
> Drizzle JDBC and some dependencies cannot find a license have been removed.
>
> 3. If there are no more questions, we will start to merge under the master
> JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to
> review the patches.
>
>
>
> Thanks,
>
> Jiajia
>
>
>
> <#m_8069559770700476617_this>*From:* Colm O hEigeartaigh [mailto:
> coheigea@apache.org]
> *Sent:* Wednesday, December 6, 2017 11:23 PM
> *To:* Li, Jiajia <ji...@intel.com>
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>;
> kerby@directory.apache.org
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Hi Jiajia,
>
> Perhaps you could get one of the Alibaba contributors to mail "
> secretary@apache.org" and ask if there is a CCLA on record?
>
> Colm.
>
>
>
> On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> I think Intel have provided the CCLA when contributing Kerby to Apache.
> But I'm not sure whether Alibaba already provided, is there one place we
> could check it?
>
>
>
> Thanks,
>
> Jiajia
>
> <#m_8069559770700476617_this>
>
> <#m_8069559770700476617_this>*From:* Colm O hEigeartaigh [mailto:
> coheigea@apache.org]
> *Sent:* Tuesday, December 5, 2017 1:50 AM
> *To:* kerby@directory.apache.org
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Do we have both CCLAs filed for Intel and Alibaba?
>
> Colm.
>
>
>
> On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
>
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
>
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
+1.
Colm.
On Fri, Dec 15, 2017 at 3:52 AM, 郑锴(铁杰) <zh...@alibaba-inc.com> wrote:
> Thanks Jiajia for the update and major progress. It would be good to see
> HAS feature consolidated into Kerby KDC offering.
>
> Regards,
> Kai
>
> ------------------------------------------------------------------
> 发件人：Li, Jiajia <ji...@intel.com>
> 发送时间：2017年12月15日(星期五) 10:38
> 收件人：Apache Directory Developers List <de...@directory.apache.org>;
> coheigea@apache.org <co...@apache.org>
> 抄 送：kerby@directory.apache.org <ke...@directory.apache.org>
> 主 题：RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi all, <#m_8069559770700476617_this>
>
>
>
> Status update:
>
>
>
> 1. I've checked both Intel and Alibaba have signed the CCLA.
>
> 2. We have fixed dependency issues: mysql JDBC driver is replaced with
> Drizzle JDBC and some dependencies cannot find a license have been removed.
>
> 3. If there are no more questions, we will start to merge under the master
> JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to
> review the patches.
>
>
>
> Thanks,
>
> Jiajia
>
>
>
> <#m_8069559770700476617_this>*From:* Colm O hEigeartaigh [mailto:
> coheigea@apache.org]
> *Sent:* Wednesday, December 6, 2017 11:23 PM
> *To:* Li, Jiajia <ji...@intel.com>
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>;
> kerby@directory.apache.org
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Hi Jiajia,
>
> Perhaps you could get one of the Alibaba contributors to mail "
> secretary@apache.org" and ask if there is a CCLA on record?
>
> Colm.
>
>
>
> On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> I think Intel have provided the CCLA when contributing Kerby to Apache.
> But I'm not sure whether Alibaba already provided, is there one place we
> could check it?
>
>
>
> Thanks,
>
> Jiajia
>
> <#m_8069559770700476617_this>
>
> <#m_8069559770700476617_this>*From:* Colm O hEigeartaigh [mailto:
> coheigea@apache.org]
> *Sent:* Tuesday, December 5, 2017 1:50 AM
> *To:* kerby@directory.apache.org
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Do we have both CCLAs filed for Intel and Alibaba?
>
> Colm.
>
>
>
> On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
>
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
>
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
回复：[DISCUSS] Merge HAS to Apache Kerby
Posted by "郑锴(铁杰)" <zh...@alibaba-inc.com>.
Thanks Jiajia for the update and major progress. It would be good to see HAS feature consolidated into Kerby KDC offering.
Regards,Kai
------------------------------------------------------------------发件人：Li, Jiajia <ji...@intel.com>发送时间：2017年12月15日(星期五) 10:38收件人：Apache Directory Developers List <de...@directory.apache.org>; coheigea@apache.org <co...@apache.org>抄 送：kerby@directory.apache.org <ke...@directory.apache.org>主 题：RE: [DISCUSS] Merge HAS to Apache Kerby
Hi all,
Status update:
1. I've checked both Intel and Alibaba have signed the CCLA.
2. We have fixed dependency issues: mysql JDBC driver is replaced with Drizzle JDBC and some dependencies cannot find a license have been removed.
3. If there are no more questions, we will start to merge under the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to review the patches.
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, December 6, 2017 11:23 PM
To: Li, Jiajia <ji...@intel.com>
Cc: Apache Directory Developers List <de...@directory.apache.org>; kerby@directory.apache.org
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "secretary@apache.org" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba
already provided, is there one place we could check it?
Thanks,
Jiajia
From:
Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to
secretary@apache.org and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org
Cc: kerby@directory.apache.org; Apache Directory Developers List <de...@directory.apache.org>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example,
the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next
step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate
tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and
server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive
some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP
that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug
in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> >
https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> >
https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
回复：[DISCUSS] Merge HAS to Apache Kerby
Posted by "郑锴(铁杰)" <zh...@alibaba-inc.com>.
Thanks Jiajia for the update and major progress. It would be good to see HAS feature consolidated into Kerby KDC offering.
Regards,Kai
------------------------------------------------------------------发件人：Li, Jiajia <ji...@intel.com>发送时间：2017年12月15日(星期五) 10:38收件人：Apache Directory Developers List <de...@directory.apache.org>; coheigea@apache.org <co...@apache.org>抄 送：kerby@directory.apache.org <ke...@directory.apache.org>主 题：RE: [DISCUSS] Merge HAS to Apache Kerby
Hi all,
Status update:
1. I've checked both Intel and Alibaba have signed the CCLA.
2. We have fixed dependency issues: mysql JDBC driver is replaced with Drizzle JDBC and some dependencies cannot find a license have been removed.
3. If there are no more questions, we will start to merge under the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to review the patches.
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, December 6, 2017 11:23 PM
To: Li, Jiajia <ji...@intel.com>
Cc: Apache Directory Developers List <de...@directory.apache.org>; kerby@directory.apache.org
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "secretary@apache.org" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba
already provided, is there one place we could check it?
Thanks,
Jiajia
From:
Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to
secretary@apache.org and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org
Cc: kerby@directory.apache.org; Apache Directory Developers List <de...@directory.apache.org>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example,
the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next
step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate
tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and
server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive
some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP
that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug
in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> >
https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> >
https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> >
https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Hi all,
Status update:
1. I've checked both Intel and Alibaba have signed the CCLA.
2. We have fixed dependency issues: mysql JDBC driver is replaced with Drizzle JDBC and some dependencies cannot find a license have been removed.
3. If there are no more questions, we will start to merge under the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to review the patches.
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, December 6, 2017 11:23 PM
To: Li, Jiajia <ji...@intel.com>
Cc: Apache Directory Developers List <de...@directory.apache.org>; kerby@directory.apache.org
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "secretary@apache.org<ma...@apache.org>" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com>> wrote:
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba already provided, is there one place we could check it?
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org<ma...@directory.apache.org>
Cc: Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com>> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to secretary@apache.org<ma...@apache.org> and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com<ma...@intel.com>]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org<ma...@apache.org>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md><http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md><http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
Hi all,
Status update:
1. I've checked both Intel and Alibaba have signed the CCLA.
2. We have fixed dependency issues: mysql JDBC driver is replaced with Drizzle JDBC and some dependencies cannot find a license have been removed.
3. If there are no more questions, we will start to merge under the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to review the patches.
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Wednesday, December 6, 2017 11:23 PM
To: Li, Jiajia <ji...@intel.com>
Cc: Apache Directory Developers List <de...@directory.apache.org>; kerby@directory.apache.org
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "secretary@apache.org<ma...@apache.org>" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com>> wrote:
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba already provided, is there one place we could check it?
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org<ma...@directory.apache.org>
Cc: Apache Directory Developers List <de...@directory.apache.org>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com>> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to secretary@apache.org<ma...@apache.org> and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com<ma...@intel.com>]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org<ma...@apache.org>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md><http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md><http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "
secretary@apache.org" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
> I think Intel have provided the CCLA when contributing Kerby to Apache.
> But I'm not sure whether Alibaba already provided, is there one place we
> could check it?
>
>
>
> Thanks,
>
> Jiajia
>
>
>
> *From:* Colm O hEigeartaigh [mailto:coheigea@apache.org]
> *Sent:* Tuesday, December 5, 2017 1:50 AM
> *To:* kerby@directory.apache.org
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Do we have both CCLAs filed for Intel and Alibaba?
>
> Colm.
>
>
>
> On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
>
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jiajia,
Perhaps you could get one of the Alibaba contributors to mail "
secretary@apache.org" and ask if there is a CCLA on record?
Colm.
On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <ji...@intel.com> wrote:
> I think Intel have provided the CCLA when contributing Kerby to Apache.
> But I'm not sure whether Alibaba already provided, is there one place we
> could check it?
>
>
>
> Thanks,
>
> Jiajia
>
>
>
> *From:* Colm O hEigeartaigh [mailto:coheigea@apache.org]
> *Sent:* Tuesday, December 5, 2017 1:50 AM
> *To:* kerby@directory.apache.org
> *Cc:* Apache Directory Developers List <de...@directory.apache.org>
> *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby
>
>
>
> Do we have both CCLAs filed for Intel and Alibaba?
>
> Colm.
>
>
>
> On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
>
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
>
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
>
> --
>
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba already provided, is there one place we could check it?
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com>> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to secretary@apache.org<ma...@apache.org> and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com<ma...@intel.com>]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org<ma...@apache.org>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md><http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md><http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: [DISCUSS] Merge HAS to Apache Kerby
Posted by "Li, Jiajia" <ji...@intel.com>.
I think Intel have provided the CCLA when contributing Kerby to Apache. But I'm not sure whether Alibaba already provided, is there one place we could check it?
Thanks,
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
Sent: Tuesday, December 5, 2017 1:50 AM
To: kerby@directory.apache.org
Cc: Apache Directory Developers List <de...@directory.apache.org>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com>> wrote:
Hi all,
Here with some status update, now all the contributors have provided the ICLA to secretary@apache.org<ma...@apache.org> and I've create the master JIRA(https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
Any more suggestions on how to merge?
Thanks,
Jiajia
From: Li, Jiajia [mailto:jiajia.li@intel.com<ma...@intel.com>]
Sent: Thursday, November 30, 2017 1:38 PM
To: coheigea@apache.org<ma...@apache.org>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache Directory Developers List <de...@directory.apache.org>>
Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
Hi Colm,
> What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
Now I understand what you mean. There are there reasons for using backend:
1. If user using the new authentication mechanism(Kerberos-based token authentication), the TGT(ticket granting ticket) could be got without backend. But TGT is not enough to access the service, after getting the TGT, next step is to get SGT(Ticket for Service), in this step, the service principal is needed in backend.
2. The new authentication mechanism is used by the end users instead of service level, services are still strongly authenticated by Kerberos, they through the keytabs to login.
3. Users or admins sometimes want to using "kinit" to get credential cache to manage the cluster, for the compatibility.
> One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
I do not know much about SecurityTokenService, from your introduction, I think STS could issue token and validate token, that is exactly the existing authentication system HAS wants to plugin, we can write the client and server plugins for STS, then using STS in HAS framework. Please correct me if I'm wrong.
We think it's more suitable to be integrated with kerby with following reasons:
1. The new authentication mechanism ("Kerberos-based token authentication") is based on the "TokenPreauth" provided in Kerby, using AuthToken to exchange a Kerberos ticket.
2. The REST APIs not only for the new authentication, also provide some useful interfaces, such as: config Kerby KDC, manage the Kerby backend, export keytab files. These could help Kerby KDC to be stronger.
3. HAS binds webserver and Kerby KDC very closely, they are all included in HasServer(we can rename it after merging), we could also think the webserver is one part of Kerby KDC, we using the webserver for KDC to receive some requests from HTTPs client.
Thanks
Jiajia
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>]
Sent: Wednesday, November 29, 2017 10:58 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Hi Jiajia,
What I meant with the point about the backend, is that it should be configurable whether to just trust the signature of the presented auth token as sufficient validation, without requiring any MySQL backend. For example, the token might be issued by an IdP that HAS "trusts", where the IdP has an identity backend of which HAS knows nothing about.
One final overall point, is that HAS looks a bit like a SecurityTokenService (STS). Apache CXF ships with a STS that I am very familiar with. It is a web application that supports a SOAP and REST interface to issue, validate tokens etc, where you can "plug in" the tokens that are supported. It might be worth exploring if the functionality of HAS could be integrated with the CXF STS.
Colm.
Thanks,
Jiajia
-----Original Message-----
From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
Sent: Tuesday, November 28, 2017 9:12 PM
To: Li, Jiajia <ji...@intel.com>>>
Cc: kerby@directory.apache.org<ma...@directory.apache.org>>; Apache Directory Developers List <de...@directory.apache.org>>>
Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
Thanks Jiajia, that was very helpful. I have some questions:
There are no HasClientPlugin implementations in the commit (unless I missed them). Is the plan to provide some later on, or is the user supposed to implement their own?
If we want to get Kerby to issue a TGT using an AuthToken currently, we have to use a token armor cache. In HAS, when it queries Kerby to get a TGT using the verified AuthToken, is this just an "internal" call so we can avoid this step?
I'm not sure why we need to verify the user information in the SQL backend.
If the received AuthToken is signed by a trusted IdP, can we not just accept the identity of the user "as is" and skip this step?
KinitTool and KinitOption in has-client-tool duplicate the Kerby versions with just a few changes. Can the changes be rolled into Kerby to prevent code duplication?
Colm.
On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <ji...@intel.com>>> wrote:
> Thanks Colm.
>
> > It sounds like a really interesting project.
> I'm glad to here that.
>
> > Have you got any feedback from the Hadoop project about it?
> We haven't proposed this solution in the hadoop community.
>
> > I'm finding it hard to understand exactly how it works though based
> > on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Following are the steps of user accessing HDFS service, taking the cmd
> "hadoop fs -ls /" as an example:
> 1. user runs the command "hadoop fs -ls /"
> 2. Hadoop client will call the "HasLoginModule",
> https://github.com/apache/directory-kerby/blob/has-
> project/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java
> 3. "HasLoginModule" will call the "HasClient",
> https://github.com/apache/
> directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> has/client/HasLoginModule.java#L237
> 4. "HasClient" will get the plugin type from config, then choose the
> right client plugin, the client plugin will collect and add some user
> info to "AuthToken", the following is the client plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the server side module.
> String getLoginType()
>
> // Perform all the client side login logics, the results wrapped in an
> AuthToken, // will be validated by HAS server.
> AuthToken login(Conf loginConf) throws HasLoginException
>
> 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> 6. After HAS server receives the message, it will call the server
> plugin, server plugin will verify the user info in AuthToken, the
> following is the server plugin interface:
>
> // Get the login module type ID, used to distinguish this module from
> others.
> // Should correspond to the client side module.
> String getLoginType()
>
> // Perform all the server side authentication logics, the results
> wrapped in an "AuthToken", // will be used to exchange a Kerberos
> ticket.
> AuthToken authenticate(AuthToken userToken) throws HasAuthenException
>
> 7. If the user info is verified in existing user authentication
> system, server plugin will return the verified "AuthToken" to Kerby
> KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> then send the TGT to HasClient through HTTPS 9. Now user login
> successful, could continue the others steps, such as:
> getting SGT ticket.
>
> We replace the step through "kinit" to get Kerberos Ticket. There are
> two important benefits:
> 1. The user's principal may not be in the backend, security admins
> won't have to migrate and sync up their user accounts to Kerberos back and forth.
> 2. Multiple users could run the job at the same time and in the same
> machine, through collecting user info from environment variables in step4.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<ma...@apache.org>>]
> Sent: Monday, November 27, 2017 6:54 PM
> To: kerby@directory.apache.org<ma...@directory.apache.org>>
> Cc: Apache Directory Developers List <de...@directory.apache.org>>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
>
> It sounds like a really interesting project. Have you got any feedback
> from the Hadoop project about it?
>
> I'm finding it hard to understand exactly how it works though based on
> the README. Could you describe how it works from a really basic point
> of view for say a simple Hadoop client? Normally I just have to use
> "kinit" to get a kerberos ticket and then I am authenticated to invoke
> on HDFS. How does HAS work differently? Where does the token pre-auth stuff fit in?
>
> Colm.
>
>
> On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <ji...@intel.com>>> wrote:
>
> > Hi all,
> >
> > I would like to post a proposal about merging a new project HAS
> > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > and Alibaba, it is a solution to support the authentication of open
> > source big data ecosystem in cloud computing platforms. I've created
> > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > Please look at
> > https://github.com/apache/directory-kerby/tree/has-project/has
> > for details.
> >
> > Background and motivation:
> > At present, the open source big data ecosystems (Hadoop/Spark) only
> > has the built-in Kerberos support on the security authentication.
> > HAS aims to build a standalone authentication service for the big
> > data ecosystem that simplifies the support of Kerberos and allows to
> > use more authentication methods.
> >
> > Targets users:
> > HAS supports various authentication mechanisms other than just
> > Kerberos, and it provides a new authentication mechanism can be easy
> > customized and plugin with existing user authentication and
> > authorization system, and security admins won't have to migrate and
> > sync up their user accounts to Kerberos back and forth.
> >
> > Architecture & Design:
> > HAS provides a new authentication mechanism ("Kerberos-based token
> > authentication"), depending on the "TokenPreauth" provided by Apache
> Kerby.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> >
> > Features:
> > 1. Provides new authentication mechanism plugin APIs to customize
> and
> > plugin with existing user authentication and authorization system.
> > Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/README.md for details.
> > 2. Provides lots of REST APIs and facility tools to simplify the
> > support of Kerberos. Kerberos is essentially a protocol, or secure
> > channel, doesn't have to be that complex to users. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/rest-api.md<http://rest-api.md><http://rest-api.md> for details.
> > 3. Provides MySQL backend for High Availability. Please look at
> > https://github.com/apache/directory-kerby/blob/has-project/
> > has/doc/mysql-backend.md<http://mysql-backend.md><http://mysql-backend.md> for details.
> > 4. New authentication mechanism now supports most of the components
> > of open source big data ecosystem with little or no changes to
> > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > look at
> > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > rt
> > s
> > for details.
> >
> > Practice
> > This solution has been deployed in Alibaba Cloud E-MapReduce production.
> >
> > Why to merge?
> > HAS provides a complete Hadoop/Spark authentication framework and
> > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > it more solid and stronger. And if HAS can be merged to Apache
> > Kerby, community will help HAS grow faster and users can more easily
> > using this solution in their own production. We have two suggestions
> > about how
> to merge:
> > - Option1:
> > Create a standalone module "kerby-has", putting HAS project under
> > this module.
> > - Option2:
> > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> >
> > Contributors:
> > Jiajia, Li (Intel)
> > Lin, Zeng (Intel)
> > Zhiqiang, Zhang (Intel)
> > Kai, Zheng (Intel)
> > Wei, Wu (Alibaba)
> > Jun, Song (Alibaba)
> > Long, Cao (Alibaba)
> > Zhenyuan, Wei (Alibaba)
> >
> > Your review efforts are truly appreciated, please feel free to
> > provide us your feedback.
> >
> > Regards,
> > Jiajia
> >
> >
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: [DISCUSS] Merge HAS to Apache Kerby
Posted by Colm O hEigeartaigh <co...@apache.org>.
Do we have both CCLAs filed for Intel and Alibaba?
Colm.
On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <ji...@intel.com> wrote:
> Hi all,
>
> Here with some status update, now all the contributors have provided the
> ICLA to secretary@apache.org and I've create the master JIRA(
> https://issues.apache.org/jira/browse/DIRKRB-671) for this merging.
> Any more suggestions on how to merge?
>
> Thanks,
> Jiajia
>
> From: Li, Jiajia [mailto:jiajia.li@intel.com]
> Sent: Thursday, November 30, 2017 1:38 PM
> To: coheigea@apache.org
> Cc: kerby@directory.apache.org; Apache Directory Developers List <
> dev@directory.apache.org>
> Subject: RE: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Colm,
>
> > What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> Now I understand what you mean. There are there reasons for using backend:
> 1. If user using the new authentication mechanism(Kerberos-based token
> authentication), the TGT(ticket granting ticket) could be got without
> backend. But TGT is not enough to access the service, after getting the
> TGT, next step is to get SGT(Ticket for Service), in this step, the service
> principal is needed in backend.
> 2. The new authentication mechanism is used by the end users instead of
> service level, services are still strongly authenticated by Kerberos, they
> through the keytabs to login.
> 3. Users or admins sometimes want to using "kinit" to get credential cache
> to manage the cluster, for the compatibility.
>
> > One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> I do not know much about SecurityTokenService, from your introduction, I
> think STS could issue token and validate token, that is exactly the
> existing authentication system HAS wants to plugin, we can write the client
> and server plugins for STS, then using STS in HAS framework. Please correct
> me if I'm wrong.
>
> We think it's more suitable to be integrated with kerby with following
> reasons:
> 1. The new authentication mechanism ("Kerberos-based token
> authentication") is based on the "TokenPreauth" provided in Kerby, using
> AuthToken to exchange a Kerberos ticket.
> 2. The REST APIs not only for the new authentication, also provide some
> useful interfaces, such as: config Kerby KDC, manage the Kerby backend,
> export keytab files. These could help Kerby KDC to be stronger.
> 3. HAS binds webserver and Kerby KDC very closely, they are all included
> in HasServer(we can rename it after merging), we could also think the
> webserver is one part of Kerby KDC, we using the webserver for KDC to
> receive some requests from HTTPs client.
>
> Thanks
> Jiajia
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org]
> Sent: Wednesday, November 29, 2017 10:58 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
>
> Hi Jiajia,
> What I meant with the point about the backend, is that it should be
> configurable whether to just trust the signature of the presented auth
> token as sufficient validation, without requiring any MySQL backend. For
> example, the token might be issued by an IdP that HAS "trusts", where the
> IdP has an identity backend of which HAS knows nothing about.
>
> One final overall point, is that HAS looks a bit like a
> SecurityTokenService (STS). Apache CXF ships with a STS that I am very
> familiar with. It is a web application that supports a SOAP and REST
> interface to issue, validate tokens etc, where you can "plug in" the tokens
> that are supported. It might be worth exploring if the functionality of HAS
> could be integrated with the CXF STS.
>
> Colm.
>
>
> Thanks,
> Jiajia
>
> -----Original Message-----
> From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> Sent: Tuesday, November 28, 2017 9:12 PM
> To: Li, Jiajia <ji...@intel.com>>
> Cc: kerby@directory.apache.org<ma...@directory.apache.org>; Apache
> Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> Thanks Jiajia, that was very helpful. I have some questions:
>
> There are no HasClientPlugin implementations in the commit (unless I
> missed them). Is the plan to provide some later on, or is the user supposed
> to implement their own?
>
> If we want to get Kerby to issue a TGT using an AuthToken currently, we
> have to use a token armor cache. In HAS, when it queries Kerby to get a TGT
> using the verified AuthToken, is this just an "internal" call so we can
> avoid this step?
>
> I'm not sure why we need to verify the user information in the SQL backend.
> If the received AuthToken is signed by a trusted IdP, can we not just
> accept the identity of the user "as is" and skip this step?
>
> KinitTool and KinitOption in has-client-tool duplicate the Kerby versions
> with just a few changes. Can the changes be rolled into Kerby to prevent
> code duplication?
>
> Colm.
>
> On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia.li@intel.com<mailto:ji
> ajia.li@intel.com>> wrote:
>
> > Thanks Colm.
> >
> > > It sounds like a really interesting project.
> > I'm glad to here that.
> >
> > > Have you got any feedback from the Hadoop project about it?
> > We haven't proposed this solution in the hadoop community.
> >
> > > I'm finding it hard to understand exactly how it works though based
> > > on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Following are the steps of user accessing HDFS service, taking the cmd
> > "hadoop fs -ls /" as an example:
> > 1. user runs the command "hadoop fs -ls /"
> > 2. Hadoop client will call the "HasLoginModule",
> > https://github.com/apache/directory-kerby/blob/has-
> > project/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java
> > 3. "HasLoginModule" will call the "HasClient",
> > https://github.com/apache/
> > directory-kerby/blob/438904f7e557a085c8c336efd2d2be
> > 304291d246/has/has-client/src/main/java/org/apache/hadoop/
> > has/client/HasLoginModule.java#L237
> > 4. "HasClient" will get the plugin type from config, then choose the
> > right client plugin, the client plugin will collect and add some user
> > info to "AuthToken", the following is the client plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the server side module.
> > String getLoginType()
> >
> > // Perform all the client side login logics, the results wrapped in an
> > AuthToken, // will be validated by HAS server.
> > AuthToken login(Conf loginConf) throws HasLoginException
> >
> > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS;
> > 6. After HAS server receives the message, it will call the server
> > plugin, server plugin will verify the user info in AuthToken, the
> > following is the server plugin interface:
> >
> > // Get the login module type ID, used to distinguish this module from
> > others.
> > // Should correspond to the client side module.
> > String getLoginType()
> >
> > // Perform all the server side authentication logics, the results
> > wrapped in an "AuthToken", // will be used to exchange a Kerberos
> > ticket.
> > AuthToken authenticate(AuthToken userToken) throws HasAuthenException
> >
> > 7. If the user info is verified in existing user authentication
> > system, server plugin will return the verified "AuthToken" to Kerby
> > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth,
> > then send the TGT to HasClient through HTTPS 9. Now user login
> > successful, could continue the others steps, such as:
> > getting SGT ticket.
> >
> > We replace the step through "kinit" to get Kerberos Ticket. There are
> > two important benefits:
> > 1. The user's principal may not be in the backend, security admins
> > won't have to migrate and sync up their user accounts to Kerberos back
> and forth.
> > 2. Multiple users could run the job at the same time and in the same
> > machine, through collecting user info from environment variables in
> step4.
> >
> >
> > Thanks,
> > Jiajia
> >
> > -----Original Message-----
> > From: Colm O hEigeartaigh [mailto:coheigea@apache.org<mailto:
> coheigea@apache.org>]
> > Sent: Monday, November 27, 2017 6:54 PM
> > To: kerby@directory.apache.org<ma...@directory.apache.org>
> > Cc: Apache Directory Developers List <dev@directory.apache.org<mailto:
> dev@directory.apache.org>>
> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby
> >
> > Hi Jiajia,
> >
> > It sounds like a really interesting project. Have you got any feedback
> > from the Hadoop project about it?
> >
> > I'm finding it hard to understand exactly how it works though based on
> > the README. Could you describe how it works from a really basic point
> > of view for say a simple Hadoop client? Normally I just have to use
> > "kinit" to get a kerberos ticket and then I am authenticated to invoke
> > on HDFS. How does HAS work differently? Where does the token pre-auth
> stuff fit in?
> >
> > Colm.
> >
> >
> > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia.li@intel.com<mailto:
> jiajia.li@intel.com>> wrote:
> >
> > > Hi all,
> > >
> > > I would like to post a proposal about merging a new project HAS
> > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel
> > > and Alibaba, it is a solution to support the authentication of open
> > > source big data ecosystem in cloud computing platforms. I've created
> > > a new branch "has-project" in Kerby, HAS is under "has" folder.
> > > Please look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has
> > > for details.
> > >
> > > Background and motivation:
> > > At present, the open source big data ecosystems (Hadoop/Spark) only
> > > has the built-in Kerberos support on the security authentication.
> > > HAS aims to build a standalone authentication service for the big
> > > data ecosystem that simplifies the support of Kerberos and allows to
> > > use more authentication methods.
> > >
> > > Targets users:
> > > HAS supports various authentication mechanisms other than just
> > > Kerberos, and it provides a new authentication mechanism can be easy
> > > customized and plugin with existing user authentication and
> > > authorization system, and security admins won't have to migrate and
> > > sync up their user accounts to Kerberos back and forth.
> > >
> > > Architecture & Design:
> > > HAS provides a new authentication mechanism ("Kerberos-based token
> > > authentication"), depending on the "TokenPreauth" provided by Apache
> > Kerby.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > >
> > > Features:
> > > 1. Provides new authentication mechanism plugin APIs to customize
> > and
> > > plugin with existing user authentication and authorization system.
> > > Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/README.md for details.
> > > 2. Provides lots of REST APIs and facility tools to simplify the
> > > support of Kerberos. Kerberos is essentially a protocol, or secure
> > > channel, doesn't have to be that complex to users. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/rest-api.md<http://rest-api.md> for details.
> > > 3. Provides MySQL backend for High Availability. Please look at
> > > https://github.com/apache/directory-kerby/blob/has-project/
> > > has/doc/mysql-backend.md<http://mysql-backend.md> for details.
> > > 4. New authentication mechanism now supports most of the
> components
> > > of open source big data ecosystem with little or no changes to
> > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please
> > > look at
> > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo
> > > rt
> > > s
> > > for details.
> > >
> > > Practice
> > > This solution has been deployed in Alibaba Cloud E-MapReduce
> production.
> > >
> > > Why to merge?
> > > HAS provides a complete Hadoop/Spark authentication framework and
> > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make
> > > it more solid and stronger. And if HAS can be merged to Apache
> > > Kerby, community will help HAS grow faster and users can more easily
> > > using this solution in their own production. We have two suggestions
> > > about how
> > to merge:
> > > - Option1:
> > > Create a standalone module "kerby-has", putting HAS project under
> > > this module.
> > > - Option2:
> > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC.
> > >
> > > Contributors:
> > > Jiajia, Li (Intel)
> > > Lin, Zeng (Intel)
> > > Zhiqiang, Zhang (Intel)
> > > Kai, Zheng (Intel)
> > > Wei, Wu (Alibaba)
> > > Jun, Song (Alibaba)
> > > Long, Cao (Alibaba)
> > > Zhenyuan, Wei (Alibaba)
> > >
> > > Your review efforts are truly appreciated, please feel free to
> > > provide us your feedback.
> > >
> > > Regards,
> > > Jiajia
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com