About Haox project

["Zheng, Kai" <ka...@intel.com>]

Hi all,

I'm Kai and a guy from Intel Big Data team. I have been working on Apache Hadoop for some while, and used to be an ApacheDS contributor and committed the feature of CredentialCahce and a simple Kinit tool, guided by Emmanuel and Kiran. After that when I was going to implement more Kerberos features I realized there needs important improvement space in the existing fundamental Kerberos codes. As a prototype I initiated Haox project  and have implemented an ASN1 encoding/decoding framework, complete encryption and checksum crypto, Kerberos codec, Keytab and CredentialCache facilities, KrbClient API and simple KDC server. It's still in its initial phase and far from ideal. However as strongly suggested by Emmanuel and Kiran, I should collaborate with ApacheDS community and continue to make the effort with you together. It's great and now I'm going to contribute the project to ApacheDS. I have discussed with Emmanuel and Kiran for quite some time about this. Based on our discussion, we have the following proposal. It's for your review and consider. Please kindly provide your feedback and advise. Thanks in advance.

1. First all our vision for the long term, we would think it makes great sense to establish a first class Kerberos implementation in Java for the Apache world targeting today's environments  in cloud, Hadoop and mobile. Therefore we could collaborate together in ApacheDS community to build a "Apache Kerberos" project. That's why we're here.

2. And as the first step to start with, ApacheDS would accept Haox as a sub-project, renamed as "Apache Kerberos". Then we consolidate existing Kerberos implementation codes in ApacheDS into the new project;

3. Afterwards we will make further effort to push "Apache Kerberos" out as a top level project, probably in a year or so.

We have discussed other approaches, like using the good stuffs in Haox to replace the corresponding parts in ApacheDS Kerberos, as one may also suggest. So let's figure out what might be good parts implemented in Haox.

1.     Haox-ASN1. It's a schema/model driven ASN1 parser and codec. Based on the framework new Kerberos types can be easily defined following fixed patterns, without having to take care of ASN1 encoding/decoding procedures. We see this as very important because Kerberos itself and related extensions are very ASN1 heavy. Without the basic friendly library support it's hard for new developers to contribute on new Kerberos evolving features. We tested the codec performance and it outperforms a little. Emmanuel has even contributed a few optimization improvements.



2.     Kerberos crypto. The crypto framework (based on RFC3961) and library implement all the encryption types and checksum types that MIT Kerberos supports, like DES, DES3, AES, RC4, CAMELLIA, with extensive interoperable tests covered. It relies on JCE for encryption but not any Kerberos facilities provided in JRE. Potentially we could port it to other non-Oracle Java environment like Android system, assuming we can implement corresponding encryption and hash providers which can be pluggable.


3.     Kerberos codec. Based on Haox-Asn1, the codec has implemented all the necessary Kerberos types defined in Krb5(RFC4210), Preauth/FAST(RFC6113), and PKINIT(4556). The core part has some test cases covered already.


4.     KDC server. The implementation abstracts an KrbIdentityService API and decouples any specific identity backend from the Kerberos server. We can provide an LDAP based backend, or allow vendor to implement their own backed by SQL database, external LDAP server or whatever. For simple unit test usage, a memory HashMap is all the needs. This allows our embedded KDC really very lightweight avoiding many unnecessary dependencies.


5.     Haox-event. It implements a UDP/TCP mixed network event support based on Java NIO and reactor pattern according to Kerberos's needs, where UDP and TCP are both required, and even in IAKERB, a KDC proxy would serve both TCP server role and UDP/TCP client role. In Haox-event, all these roles can be done in an unified event receiving, processing and replying. Test cases are given but it definitely need to tune and improve. Again, it doesn't rely on any other libraries, thus based on it KrbClient could easily be supported in other Java environment.

Keeping our long term goal in mind, we thought above functionalities, properties and considerations may serve better to establish the foundation for "Apache Kerberos" project, and it would be much easier to consolidate the existing Kerberos implementation into it.

So as a first step for the collaboration, how about:

1.     Prepare, which involves:

1) Necessary cleanup;

2) All Kerberos related codes will start package names with "org.apache.kerberos", all the non-Kerberos related codes will be moved into contrib projects or libraries, like Haox-ASN1, Haox-event, Haox-config, which are actually not Kerberos logic specific.

2. Move Haox to be an ApacheDS sub-project:
1) Renaming Haox as "Apache Kerberos";
2) A separate GIT repo;
3) A separate coding style, subject to be discussed and determined.

Then as the 2nd step, consolidate the necessary parts in existing Kerberos implementation. Below is the initial list and please complement:

1.     Apache Directory Server support. We need to create an LdapIdentityBackend using Apache Directory Server and use it by default, embedded in the KDC server.

2.     Installation packaging and service wrapper. Haox hasn't implemented such but ApacheDS does.

3.     Tests. All the Kerberos related tests should be evaluated and ported to the new codebase if not obsolete.

In addition to above, we also need to consider the following aspects, to make it good enough for potential users.

1.     Adding the missing Javadoc.

2.     Adding more tests.

3.     Adding the documentation.
4. Building the web site

With such done, then we might obsolete the AacheDS Kerberos implementation and might move all the related codes into an history folder for existing users. We would encourage users to use the new project.
For the 3rd step, we would continue to implement some essential and advanced features, which may include:

1.     Cross realm support, which means Apache Kerberos can federate with MIT Kerberos or MS AD;

2.     Full functional client tools, like kinit, klist and kadmin, or other alternatives;

3.     Preauth/FAST framework, which allows to support other authentication mechanisms based on the secure channel protected by an Armor TGT/key;

4.     PKINIT, authenticating user using x509 certificate other than plain password;

5.     Token preauth, authenticating user using a JWT token, to support OAuth.

6.     ...

Note tasks in step2 and 3 are not necessarily to be done in the order. We're likely to work on them in parallel.

We would encourage more developers to contribute to the effort and develop the community. Without that, we may not be able to make it to have a high quality of Kerberos implementation, with all the important Kerberos feature gaps being filled.

Then finally as the last step, in some appropriate time we would push Apache Kerberos out to be an Apache top level project.

All the points are just suggestions and subject to your discussion. Your feedback is most welcome. As widely discussed in a week or so, this proposal is to be finalized and then re-posted. Then let's collaborate and make it together.

Thanks for your support. Below is Haox project.
https://github.com/drankye/haox

At last but not least, many thanks to Emmanuel and Kiran. I can't appreciate them anymore. Without their encouraging, I would not have the motivation though it's still far away from the long term goal of "a first class Kerberos implementation". I do believe the Java world needs such, and the whole Apache world needs it as well. Thanks for your great questions, suggestions, complement points to sort this out. Thanks for helping me understand further about the Apache way, particularly regarding how I should proceed and collaborate with you guys in the community.

Thank you all.

Regards,
Kai

RE: About Haox project

["Zheng, Kai" <ka...@intel.com>]

Hi Kiran,

Thanks for your help and feedback. Yes as I discussed with you and Emmanuel and maybe not explicitly said here, I was inspired and encouraged by the saying about “a first class Kerberos implementation” in Java for the Apache world. What I would add to this is: 1) let it standalone apart from ApacheDS in the future so that the Kerberos implementation can evolve independently; 2) work on it to meet with today’s challenges in new environments majorly in cloud, Hadoop and mobile. Kerberos itself is always ongoing and evolving. We need to complete and be updated.

Yes I agree with your points about the crypto part. I forgot to mention that I implemented the codes majorly by referencing the codes in MIT Kerberos. You might notice this already from the framework and API (similarly for another part, the preauth plugin framework). I spent a lot of time on that part. It’s not easy and I was going to be crazy frankly speaking. I almost gave up implementing it by myself and would have to leverage existing ones like from JRE and MIT Kerberos library (thru JNI), but I realized we need our own hacks otherwise we have to rely on the involved dependencies. For example, a lightweight Kerberos client for smart phones isn’t going to be possible. If we get this part refined and done in well sense, then it will be easy for us to support more client libraries in other languages by just porting it, in the long term future.

Regards,
Kai

From: Kiran Ayyagari [mailto:kayyagari@apache.org]
Sent: Wednesday, December 24, 2014 7:36 AM
To: Apache Directory Developers List
Subject: Re: About Haox project

Welcome Kai,

On Wed, Dec 24, 2014 at 1:57 AM, Zheng, Kai <ka...@intel.com>> wrote:
Hi all,

I’m Kai and a guy from Intel Big Data team. I have been working on Apache Hadoop for some while, and used to be an ApacheDS contributor and committed the feature of CredentialCahce and a simple Kinit tool, guided by Emmanuel and Kiran. After that when I was going to implement more Kerberos features I realized there needs important improvement space in the existing fundamental Kerberos codes. As a prototype I initiated Haox project  and have implemented an ASN1 encoding/decoding framework, complete encryption and checksum crypto, Kerberos codec, Keytab and CredentialCache facilities, KrbClient API and simple KDC server. It’s still in its initial phase and far from ideal. However as strongly suggested by Emmanuel and Kiran, I should collaborate with ApacheDS community and continue to make the effort with you together. It’s great and now I’m going to contribute the project to ApacheDS. I have discussed with Emmanuel and Kiran for quite some time about this. Based on our discussion, we have the following proposal. It’s for your review and consider. Please kindly provide your feedback and advise. Thanks in advance.

1. First all our vision for the long term, we would think it makes great sense to establish a first class Kerberos implementation in Java for the Apache world targeting today’s environments  in cloud, Hadoop and mobile. Therefore we could collaborate together in ApacheDS community to build a “Apache Kerberos” project. That’s why we’re here.

it was indeed and still is a goal of ApacheDS
2. And as the first step to start with, ApacheDS would accept Haox as a sub-project, renamed as “Apache Kerberos”. Then we consolidate existing Kerberos implementation codes in ApacheDS into the new project;

and I hope that goal can be reached now with this effort
3. Afterwards we will make further effort to push “Apache Kerberos” out as a top level project, probably in a year or so.

sure it can be, MINA became a TLP this way
We have discussed other approaches, like using the good stuffs in Haox to replace the corresponding parts in ApacheDS Kerberos, as one may also suggest. So let’s figure out what might be good parts implemented in Haox.

1.     Haox-ASN1. It’s a schema/model driven ASN1 parser and codec. Based on the framework new Kerberos types can be easily defined following fixed patterns, without having to take care of ASN1 encoding/decoding procedures. We see this as very important because Kerberos itself and related extensions are very ASN1 heavy. Without the basic friendly library support it’s hard for new developers to contribute on new Kerberos evolving features. We tested the codec performance and it outperforms a little. Emmanuel has even contributed a few optimization improvements.



2.     Kerberos crypto. The crypto framework (based on RFC3961) and library implement all the encryption types and checksum types that MIT Kerberos supports, like DES, DES3, AES, RC4, CAMELLIA, with extensive interoperable tests covered. It relies on JCE for encryption but not any Kerberos facilities provided in JRE. Potentially we could port it to other non-Oracle Java environment like Android system, assuming we can implement corresponding encryption and hash providers which can be pluggable.


3.     Kerberos codec. Based on Haox-Asn1, the codec has implemented all the necessary Kerberos types defined in Krb5(RFC4210), Preauth/FAST(RFC6113), and PKINIT(4556). The core part has some test cases covered already.


4.     KDC server. The implementation abstracts an KrbIdentityService API and decouples any specific identity backend from the Kerberos server. We can provide an LDAP based backend, or allow vendor to implement their own backed by SQL database, external LDAP server or whatever. For simple unit test usage, a memory HashMap is all the needs. This allows our embedded KDC really very lightweight avoiding many unnecessary dependencies.


5.     Haox-event. It implements a UDP/TCP mixed network event support based on Java NIO and reactor pattern according to Kerberos’s needs, where UDP and TCP are both required, and even in IAKERB, a KDC proxy would serve both TCP server role and UDP/TCP client role. In Haox-event, all these roles can be done in an unified event receiving, processing and replying. Test cases are given but it definitely need to tune and improve. Again, it doesn’t rely on any other libraries, thus based on it KrbClient could easily be supported in other Java environment.

Keeping our long term goal in mind, we thought above functionalities, properties and considerations may serve better to establish the foundation for “Apache Kerberos” project, and it would be much easier to consolidate the existing Kerberos implementation into it.

So as a first step for the collaboration, how about:

1.     Prepare, which involves:

1) Necessary cleanup;

2) All Kerberos related codes will start package names with “org.apache.kerberos”, all the non-Kerberos related codes will be moved into contrib projects or libraries, like Haox-ASN1, Haox-event, Haox-config, which are actually not Kerberos logic specific.

2. Move Haox to be an ApacheDS sub-project:
1) Renaming Haox as “Apache Kerberos”;
2) A separate GIT repo;
3) A separate coding style, subject to be discussed and determined.

Then as the 2nd step, consolidate the necessary parts in existing Kerberos implementation. Below is the initial list and please complement:

1.     Apache Directory Server support. We need to create an LdapIdentityBackend using Apache Directory Server and use it by default, embedded in the KDC server.

2.     Installation packaging and service wrapper. Haox hasn’t implemented such but ApacheDS does.

3.     Tests. All the Kerberos related tests should be evaluated and ported to the new codebase if not obsolete.

In addition to above, we also need to consider the following aspects, to make it good enough for potential users.

1.     Adding the missing Javadoc.

2.     Adding more tests.

3.     Adding the documentation.
4. Building the web site

With such done, then we might obsolete the AacheDS Kerberos implementation and might move all the related codes into an history folder for existing users. We would encourage users to use the new project.
For the 3rd step, we would continue to implement some essential and advanced features, which may include:

1.     Cross realm support, which means Apache Kerberos can federate with MIT Kerberos or MS AD;

2.     Full functional client tools, like kinit, klist and kadmin, or other alternatives;

3.     Preauth/FAST framework, which allows to support other authentication mechanisms based on the secure channel protected by an Armor TGT/key;

4.     PKINIT, authenticating user using x509 certificate other than plain password;

5.     Token preauth, authenticating user using a JWT token, to support OAuth.

6.     …

Note tasks in step2 and 3 are not necessarily to be done in the order. We’re likely to work on them in parallel.

We would encourage more developers to contribute to the effort and develop the community. Without that, we may not be able to make it to have a high quality of Kerberos implementation, with all the important Kerberos feature gaps being filled.

Then finally as the last step, in some appropriate time we would push Apache Kerberos out to be an Apache top level project.

sounds good on all parts, but as Stefan pointed out we must carefully evaluate the crypto part.
All the points are just suggestions and subject to your discussion. Your feedback is most welcome. As widely discussed in a week or so, this proposal is to be finalized and then re-posted. Then let’s collaborate and make it together.

Thanks for your support. Below is Haox project.
https://github.com/drankye/haox

At last but not least, many thanks to Emmanuel and Kiran. I can’t appreciate them anymore. Without their encouraging, I would not have the motivation though it’s still far away from the long term goal of “a first class Kerberos implementation”. I do believe the Java world needs such, and the whole Apache world needs it as well. Thanks for your great questions, suggestions, complement points to sort this out. Thanks for helping me understand further about the Apache way, particularly regarding how I should proceed and collaborate with you guys in the community.

Thank you all.

thanks for the detailed proposal, appreciate it.
Regards,
Kai



--
Kiran Ayyagari
http://keydap.com

Re: About Haox project

[Kiran Ayyagari <ka...@apache.org>]

Welcome Kai,


On Wed, Dec 24, 2014 at 1:57 AM, Zheng, Kai <ka...@intel.com> wrote:

>  Hi all,
>
>
>
> I’m Kai and a guy from Intel Big Data team. I have been working on Apache
> Hadoop for some while, and used to be an ApacheDS contributor and committed
> the feature of CredentialCahce and a simple Kinit tool, guided by Emmanuel
> and Kiran. After that when I was going to implement more Kerberos features
> I realized there needs important improvement space in the existing
> fundamental Kerberos codes. As a prototype I initiated Haox project  and
> have implemented an ASN1 encoding/decoding framework, complete encryption
> and checksum crypto, Kerberos codec, Keytab and CredentialCache facilities,
> KrbClient API and simple KDC server. It’s still in its initial phase and
> far from ideal. However as strongly suggested by Emmanuel and Kiran, I
> should collaborate with ApacheDS community and continue to make the effort
> with you together. It’s great and now I’m going to contribute the project
> to ApacheDS. I have discussed with Emmanuel and Kiran for quite some time
> about this. Based on our discussion, we have the following proposal. It’s
> for your review and consider. Please kindly provide your feedback and
> advise. Thanks in advance.
>
>
>
> 1. First all our vision for the long term, we would think it makes great
> sense to establish a first class Kerberos implementation in Java for the
> Apache world targeting today’s environments  in cloud, Hadoop and mobile.
> Therefore we could collaborate together in ApacheDS community to build a
> “Apache Kerberos” project. That’s why we’re here.
>
>
>
it was indeed and still is a goal of ApacheDS

>  2. And as the first step to start with, ApacheDS would accept Haox as a
> sub-project, renamed as “Apache Kerberos”. Then we consolidate existing
> Kerberos implementation codes in ApacheDS into the new project;
>
>
>
and I hope that goal can be reached now with this effort

>  3. Afterwards we will make further effort to push “Apache Kerberos” out
> as a top level project, probably in a year or so.
>
>
>
sure it can be, MINA became a TLP this way

>  We have discussed other approaches, like using the good stuffs in Haox
> to replace the corresponding parts in ApacheDS Kerberos, as one may also
> suggest. So let’s figure out what might be good parts implemented in Haox.
>
> 1.     Haox-ASN1. It’s a schema/model driven ASN1 parser and codec. Based
> on the framework new Kerberos types can be easily defined following fixed
> patterns, without having to take care of ASN1 encoding/decoding procedures.
> We see this as very important because Kerberos itself and related
> extensions are very ASN1 heavy. Without the basic friendly library support
> it’s hard for new developers to contribute on new Kerberos evolving
> features. We tested the codec performance and it outperforms a little.
> Emmanuel has even contributed a few optimization improvements.
>
>
>
> 2.     Kerberos crypto. The crypto framework (based on RFC3961) and
> library implement all the encryption types and checksum types that MIT
> Kerberos supports, like DES, DES3, AES, RC4, CAMELLIA, with extensive
> interoperable tests covered. It relies on JCE for encryption but not any
> Kerberos facilities provided in JRE. Potentially we could port it to other
> non-Oracle Java environment like Android system, assuming we can implement
> corresponding encryption and hash providers which can be pluggable.
>
>
>
> 3.     Kerberos codec. Based on Haox-Asn1, the codec has implemented all
> the necessary Kerberos types defined in Krb5(RFC4210),
> Preauth/FAST(RFC6113), and PKINIT(4556). The core part has some test cases
> covered already.
>
>
>
> 4.     KDC server. The implementation abstracts an KrbIdentityService API
> and decouples any specific identity backend from the Kerberos server. We
> can provide an LDAP based backend, or allow vendor to implement their own
> backed by SQL database, external LDAP server or whatever. For simple unit
> test usage, a memory HashMap is all the needs. This allows our embedded KDC
> really very lightweight avoiding many unnecessary dependencies.
>
>
>
> 5.     Haox-event. It implements a UDP/TCP mixed network event support
> based on Java NIO and reactor pattern according to Kerberos’s needs, where
> UDP and TCP are both required, and even in IAKERB, a KDC proxy would serve
> both TCP server role and UDP/TCP client role. In Haox-event, all these
> roles can be done in an unified event receiving, processing and replying.
> Test cases are given but it definitely need to tune and improve. Again, it
> doesn’t rely on any other libraries, thus based on it KrbClient could
> easily be supported in other Java environment.
>
>
>
> Keeping our long term goal in mind, we thought above functionalities,
> properties and considerations may serve better to establish the foundation
> for “Apache Kerberos” project, and it would be much easier to consolidate
> the existing Kerberos implementation into it.
>
>
>
> So as a first step for the collaboration, how about:
>
> 1.     Prepare, which involves:
>
> 1) Necessary cleanup;
>
> 2) All Kerberos related codes will start package names with
> “org.apache.kerberos”, all the non-Kerberos related codes will be moved
> into contrib projects or libraries, like Haox-ASN1, Haox-event,
> Haox-config, which are actually not Kerberos logic specific.
>
>
>
> 2. Move Haox to be an ApacheDS sub-project:
>
> 1) Renaming Haox as “Apache Kerberos”;
>
> 2) A separate GIT repo;
>
> 3) A separate coding style, subject to be discussed and determined.
>
>
>
> Then as the 2nd step, consolidate the necessary parts in existing Kerberos
> implementation. Below is the initial list and please complement:
>
> 1.     Apache Directory Server support. We need to create an
> LdapIdentityBackend using Apache Directory Server and use it by default,
> embedded in the KDC server.
>
> 2.     Installation packaging and service wrapper. Haox hasn’t
> implemented such but ApacheDS does.
>
> 3.     Tests. All the Kerberos related tests should be evaluated and
> ported to the new codebase if not obsolete.
>
>
>
> In addition to above, we also need to consider the following aspects, to
> make it good enough for potential users.
>
> 1.     Adding the missing Javadoc.
>
> 2.     Adding more tests.
>
> 3.     Adding the documentation.
>
> 4. Building the web site
>
>
>
> With such done, then we might obsolete the AacheDS Kerberos implementation
> and might move all the related codes into an history folder for existing
> users. We would encourage users to use the new project.
>
>  For the 3rd step, we would continue to implement some essential and
> advanced features, which may include:
>
> 1.     Cross realm support, which means Apache Kerberos can federate with
> MIT Kerberos or MS AD;
>
> 2.     Full functional client tools, like kinit, klist and kadmin, or
> other alternatives;
>
> 3.     Preauth/FAST framework, which allows to support other
> authentication mechanisms based on the secure channel protected by an Armor
> TGT/key;
>
> 4.     PKINIT, authenticating user using x509 certificate other than
> plain password;
>
> 5.     Token preauth, authenticating user using a JWT token, to support
> OAuth.
>
> 6.     …
>
>
>
> Note tasks in step2 and 3 are not necessarily to be done in the order.
> We’re likely to work on them in parallel.
>
>
>
> We would encourage more developers to contribute to the effort and develop
> the community. Without that, we may not be able to make it to have a high
> quality of Kerberos implementation, with all the important Kerberos feature
> gaps being filled.
>
>
>
> Then finally as the last step, in some appropriate time we would push
> Apache Kerberos out to be an Apache top level project.
>
>
>
sounds good on all parts, but as Stefan pointed out we must carefully
evaluate the crypto part.

>  All the points are just suggestions and subject to your discussion. Your
> feedback is most welcome. As widely discussed in a week or so, this
> proposal is to be finalized and then re-posted. Then let’s collaborate and
> make it together.
>
>
>
> Thanks for your support. Below is Haox project.
>
> https://github.com/drankye/haox
>
>
>
> At last but not least, many thanks to Emmanuel and Kiran. I can’t
> appreciate them anymore. Without their encouraging, I would not have the
> motivation though it’s still far away from the long term goal of “a first
> class Kerberos implementation”. I do believe the Java world needs such, and
> the whole Apache world needs it as well. Thanks for your great questions,
> suggestions, complement points to sort this out. Thanks for helping me
> understand further about the Apache way, particularly regarding how I
> should proceed and collaborate with you guys in the community.
>
>
>
> Thank you all.
>
>
>
thanks for the detailed proposal, appreciate it.

>  Regards,
>
> Kai
>



-- 
Kiran Ayyagari
http://keydap.com

RE: About Haox project

["Zheng, Kai" <ka...@intel.com>]

Hi Emmanuel,

Yes I understand. As I replied to Stefan, I will clean up and ensure this. Frankly speaking, I'm running into this a little too early. Haox is far from ideal yet. 

The following two files in your question are really not relevant to this effort and I will remove them.
GssKrb5ServerExt.java
GssKrb5BaseExt.java

The following file is from http://www.java2s.com/Code/Java/Security/ImplementstheMD4messagedigestalgorithminJava.htm, as the comment tells. Will think about how to replace it.
Md4.java

The kerb-core-test was originated/adapted from JaasLounge project and I guess I have to remove the whole as well.

Regards,
Kai

-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com] 
Sent: Wednesday, December 24, 2014 3:56 PM
To: Apache Directory Developers List
Subject: Re: About Haox project

Le 23/12/14 20:07, Stefan Seelmann a écrit :
> * Two source files contain header with Oracle copyright and GPL license.
> This code needs to be replaced. I hope no other code was taken from 
> JDK sources.

I want to be very clear here : copying existing code found on the internet is an ABSOLUTE no go. Even transforming some code that someone found on the internet is *NOT* permitted. In practice, the way to go is simply *NOT* to even open a file from code under a GPL or private license, if you want to implement some feature. Doing so would taint your code, and expose The ASF to potential legal threats.

Re: About Haox project

[Emmanuel Lécharny <el...@gmail.com>]

Le 23/12/14 20:07, Stefan Seelmann a écrit :
> * Two source files contain header with Oracle copyright and GPL license.
> This code needs to be replaced. I hope no other code was taken from JDK
> sources.

I want to be very clear here : copying existing code found on the
internet is an ABSOLUTE no go. Even transforming some code that someone
found on the internet is *NOT* permitted. In practice, the way to go is
simply *NOT* to even open a file from code under a GPL or private
license, if you want to implement some feature. Doing so would taint
your code, and expose The ASF to potential legal threats.

RE: About Haox project

["Zheng, Kai" <ka...@intel.com>]

Hi Stefan,

Thanks for your welcome and great feedback.

>> That is a huge plan and vision, I prefer to do smaller steps, but good luck
Yes I quite agree. This plan in the vision might be helpful to serve as a guidance or indication for the long term effort. Itself won't happen or even make any sense without concrete phases and steps. I would just keep the vision in mind. Would you think it works or not having the sub-project using the codebase we work on the implementation incrementally (smaller steps)? It could be easier, not having to impact Haox violently and break existing users of ApacheDS Kerberos. Initially, the sub-project is rather like a incubating project and allows to evolve independently just as other sub-projects do. Meanwhile, ApacheDS itself can decouple from Kerberos yes again in smaller steps. Then in some time we get to a phase, like mentioned in the step 2 in the proposal.

>> This proposal also means to separate all the Kerberos functionality out of ApacheDS so that it "only" remains an LDAP server? I'd support that, this would reduce complexity, modularization is good.
Yes, I thought so. Looks like we all love this. It's discussion result with Emmanuel and Kiran. 

>> Security nightmare. When implementing crypto yourself you need to know what you are doing and need to react on security issues.
I agree. The codes in that part would definitely need more attention and review. But I would clarify that the Kerberos crypto doesn't implement its own underlying encryption and MAC algorithms, it has to rely on existing like JCE. In its layer we follow the related spec and make the resultant encryption types and checksum types compatible and interoperable with MIT Kerberos. Kerberos itself is all about security, the crypto is the core part I guess so I understand your concern. It's possible we use the API and implement a JRE backed crypto. A note is the Kerberos crypto in JRE isn't complete, like lacking CAMELLIA related etypes. It only works on server side. On client side if we would support other Java environment, we would need our own tweaks. We need security experts from the community, Haox is just a beginning, as I said it's far from ideal. We would have a long journey.

>> Two source files contain header with Oracle copyright and GPL license.
>>This code needs to be replaced. I hope no other code was taken from JDK sources.
Yes, I will clean up such and ensure this. 

Regards,
Kai

-----Original Message-----
From: Stefan Seelmann [mailto:mail@stefan-seelmann.de] 
Sent: Wednesday, December 24, 2014 3:07 AM
To: dev@directory.apache.org
Subject: Re: About Haox project

Hi,

first a warm welcome :)

Just a few first notes, need more time to read the mail again...

* That is a huge plan and vision, I prefer to do smaller steps, but good luck

* This proposal also means to separate all the Kerberos functionality out of ApacheDS so that it "only" remains an LDAP server? I'd support that, this would reduce complexity, modularization is good.

* Security nightmare. When implementing crypto yourself you need to know what you are doing and need to react on security issues.

* Two source files contain header with Oracle copyright and GPL license.
This code needs to be replaced. I hope no other code was taken from JDK sources.

Kind Regards,
Stefan

Re: About Haox project

[Stefan Seelmann <ma...@stefan-seelmann.de>]

Hi,

first a warm welcome :)

Just a few first notes, need more time to read the mail again...

* That is a huge plan and vision, I prefer to do smaller steps, but good
luck

* This proposal also means to separate all the Kerberos functionality
out of ApacheDS so that it "only" remains an LDAP server? I'd support
that, this would reduce complexity, modularization is good.

* Security nightmare. When implementing crypto yourself you need to know
what you are doing and need to react on security issues.

* Two source files contain header with Oracle copyright and GPL license.
This code needs to be replaced. I hope no other code was taken from JDK
sources.

Kind Regards,
Stefan

RE: About Haox project

["Zheng, Kai" <ka...@intel.com>]

Hi Pierre,

Thanks for your welcome and feedback!

>>Apart from the technical stuff, the most important is aspect addressing the legal risks that incorporating the Haox project brings. Some have already been identified earlier. Some will pop up during the on-boarding process
Yes I agree. I’m working on it as my higher priority. In fact if you’d like to check my latest commits in Haox project, it’s all about clearance. I’m reviewing every piece of the codes as Emmanuel required to make sure they’re all fine. I would not say it will take too long as most of the codes are written by myself, and it’s not hard for me to get that aspect resolved.

>> Though I see benefits and hurdles that can be overcome, I would like to see more viewpoints.
Yes more feedback would be welcome.

>> Because I also see an increase in workload in the near future (when a positive outcome will establish itself). And this increase will put pressure on our short term plans and activities. We need to address that too.
I agree. It does involve some workload. Besides the obvious workload brought by the sub project, the directory server will also need to decouple from Kerberos aspect, as Kiran and Emmanuel discussed with me before.

Regards,
Kai

From: Pierre Smits [mailto:pierre.smits@gmail.com]
Sent: Saturday, December 27, 2014 9:06 AM
To: Apache Directory Developers List
Subject: Re: About Haox project

Kai, All,

First of all, a hearty welcome and thanks for reaching out to this community. Plus, though a bit delayed, all the seasons greetings to each and everyone.

Having read the initial post and the reactions by the others, I can only concur - from the limited understanding I have regarding the technical aspects - that there seems to be a fit within this project.

Yes, there is overlapping and complementing code between the Haox project and (several of) our sub projects. For sure, the overlap and the other issues mentioned are hurdles that can be overcome. And the complementing aspects within Hoax and our solutions will not only strengthen as well as simplifying the individual solutions.

I also believe, given the expressions in the postings in this thread, that there is a cultural fit. Having said all that, there is a lot of work to be done after a consensus has been reached. Apart from the technical stuff, the most important is aspect addressing the legal risks that incorporating the Haox project brings. Some have already been identified earlier. Some will pop up during the on-boarding process.
Though I see benefits and hurdles that can be overcome, I would like to see more viewpoints.
Because I also see an increase in workload in the near future (when a positive outcome will establish itself). And this increase will put pressure on our short term plans and activities. We need to address that too.

Best regards,

Pierre Smits

ORRTIZ.COM<http://www.orrtiz.com>
Services & Solutions for Cloud-
Based Manufacturing, Professional
Services and Retail & Trade
http://www.orrtiz.com<http://www.orrtiz.com/>

Re: About Haox project

[Pierre Smits <pi...@gmail.com>]

Kai, All,

First of all, a hearty welcome and thanks for reaching out to this
community. Plus, though a bit delayed, all the seasons greetings to each
and everyone.

Having read the initial post and the reactions by the others, I can only
concur - from the limited understanding I have regarding the technical
aspects - that there seems to be a fit within this project.

Yes, there is overlapping and complementing code between the Haox project
and (several of) our sub projects. For sure, the overlap and the other
issues mentioned are hurdles that can be overcome. And the complementing
aspects within Hoax and our solutions will not only strengthen as well as
simplifying the individual solutions.

I also believe, given the expressions in the postings in this thread, that
there is a cultural fit. Having said all that, there is a lot of work to be
done after a consensus has been reached. Apart from the technical stuff,
the most important is aspect addressing the legal risks that incorporating
the Haox project brings. Some have already been identified earlier. Some
will pop up during the on-boarding process.

Though I see benefits and hurdles that can be overcome, I would like to see
more viewpoints.
Because I also see an increase in workload in the near future (when a
positive outcome will establish itself). And this increase will put
pressure on our short term plans and activities. We need to address that
too.

Best regards,

Pierre Smits

*ORRTIZ.COM <http://www.orrtiz.com>*
Services & Solutions for Cloud-
Based Manufacturing, Professional
Services and Retail & Trade
http://www.orrtiz.com

RE: About Haox project

["Zheng, Kai" <ka...@intel.com>]

Sorry I should have mentioned it as "Kerberos cryptosystem", as RFC3961 says. Thus it could avoid the confusion here since crypto often means encryption crypto.
http://tools.ietf.org/html/rfc3961.html

To clarify further and also in case it helps.
1. A Kerberos cryptosystem may take care of the following aspects:
1) How user plain password to be transformed into Kerberos encryption key;
2) How random bytes to be transformed into Kerberos encryption key;
3) How Kerberos message to be reorganized into cipher buffer to be encrypted, like padding, adding confounder, and etc.; similarly for decryption;
4) How Kerberos message to be computed for a checksum and verified against the checksum.

2. MIT Kerberos supports encryption types as the following doc tells. The kerb-crypto in Haox is totally following it.
http://web.mit.edu/Kerberos/krb5-1.11/doc/admin/conf_files/kdc_conf.html#encryption-types


>>That's not that simple. Of course, we should support MIT based configuration, but you also have to consider those admin who have to install a kerberos server and don't know about tjose files : they most certainly expect the product we deliver to take care of the config without having to deal with .conf files...
Yes I agree. We'll need a JIRA entry to fully discuss and document how we would support configuration, how we can simplify the administration work and avoid complexities here. 

Regards,
Kai

-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com] 
Sent: Saturday, December 27, 2014 3:39 PM
To: Apache Directory Developers List
Subject: Re: About Haox project

Le 27/12/14 02:36, Zheng, Kai a écrit :
>
>>> I wanted to build a ASN/1 compiler (see 
>>> http://svn.apache.org/repos/asf/labs/dungeon/)
>>> Implementing a new Codec requires a huge effort, even for me.
> It's very interesting about the compiler. Actually it was in my mind to support such compiler for haox-asn1 library, enabling to compile an ASN1 definition module (like krb5.asn1) and automatically generate all the necessary Java types (like Kerberos codec classes), and we don't even bother to code them manually.  Yes it's hard, huge effort is involved, not easy for me either.

The major pb with this compiler is that we have to be able to process teh full ASN.1 grammar, and it's quite complex. We also have to think about the generator, and we don't have only one target : DER/CER/BER is simple, XER is probably useless, but PER is mandatory and this is really the complex one. All in all, this is not simple...

>
>>> I would rather first check what Bouncy Castle has to propose in this area. I'd be really careful here, implementing some crypto amgorithm is seriously complex, and require a lot of care.
> Sorry I'm confusing and have to be clear here. When I mentioned "implement all the encryption types and checksum types that MIT Kerberos supports, like DES, DES3, AES, RC4, CAMELLIA", I mean we need to implement the Kerberos encryption types and checksum types which are often mentioned as Kerberos crypto. Such crypto isn't the one in JCE neither any in BC, where only covers concrete encryption algorithms. Kerberos crypto is based on concrete encryption algorithms and implementations, but would never bother to define and implement any encryption algorithms itself.
>
> Yes we could rely on the Kerberos crypto implemented in JRE, as existing ApacheDS Kerberos implementation does. The problems are: 1) JRE doesn't expose the functionalities fully and is very limited; 2) It's not complete and we can't guarantee full interoperable with MIT Kerberos and MS-AD; 3) Relying on JRE means we'll never be able to go beyond that, if the corresponding support isn't available in JRE, then we'll stop there. You might agree that many modern Kerberos features are just not available yet.
>
> In my view we have to tweak it for ourselves, that's why I spent so much time on it. The framework originates the one from MIT Kerberos and is very flexible, say once a new encryption type is up then it's easy to add it accordingly so we are updated! The support is complete and all the ones provided in MIT Kerberos are done. I'm confident with it, basically I would think it's the core part that's essential for a Kerberos implementation. Without it, maybe it's fine for us to just replace the Kerberos codec part based on existing Kerberos implementation. Yes we must be careful with the codes. I would be very happy for your review of it and make it strong enough. 

Ok, seems like I have to do my homework in this area ;-)


>>> has also to be considered is the configuration part, and integration with Studio.
> Yes agree. Regarding Kerberos configuration either in client side (krb5.conf) or server side (kdc.conf), I guess we'd better make it compatible with MIT Kerberos so that administrators won't have to learn about it. Also in our documentation for simple we would just say "please reference MIT Krb5 doc" as we're going the same way. A contributor is work on it and will commit the codes once the codebase is imported.
That's not that simple. Of course, we should support MIT based configuration, but you also have to consider those admin who have to install a kerberos server and don't know about tjose files : they most certainly expect the product we deliver to take care of the config without having to deal with .conf files...

Re: About Haox project

[Emmanuel Lécharny <el...@gmail.com>]

Le 27/12/14 02:36, Zheng, Kai a écrit :
>
>>> I wanted to build a ASN/1 compiler (see http://svn.apache.org/repos/asf/labs/dungeon/)
>>> Implementing a new Codec requires a huge effort, even for me.
> It's very interesting about the compiler. Actually it was in my mind to support such compiler for haox-asn1 library, enabling to compile an ASN1 definition module (like krb5.asn1) and automatically generate all the necessary Java types (like Kerberos codec classes), and we don't even bother to code them manually.  Yes it's hard, huge effort is involved, not easy for me either.

The major pb with this compiler is that we have to be able to process
teh full ASN.1 grammar, and it's quite complex. We also have to think
about the generator, and we don't have only one target : DER/CER/BER is
simple, XER is probably useless, but PER is mandatory and this is really
the complex one. All in all, this is not simple...

>
>>> I would rather first check what Bouncy Castle has to propose in this area. I'd be really careful here, implementing some crypto amgorithm is seriously complex, and require a lot of care.
> Sorry I'm confusing and have to be clear here. When I mentioned "implement all the encryption types and checksum types that MIT Kerberos supports, like DES, DES3, AES, RC4, CAMELLIA", I mean we need to implement the Kerberos encryption types and checksum types which are often mentioned as Kerberos crypto. Such crypto isn't the one in JCE neither any in BC, where only covers concrete encryption algorithms. Kerberos crypto is based on concrete encryption algorithms and implementations, but would never bother to define and implement any encryption algorithms itself.
>
> Yes we could rely on the Kerberos crypto implemented in JRE, as existing ApacheDS Kerberos implementation does. The problems are: 1) JRE doesn't expose the functionalities fully and is very limited; 2) It's not complete and we can't guarantee full interoperable with MIT Kerberos and MS-AD; 3) Relying on JRE means we'll never be able to go beyond that, if the corresponding support isn't available in JRE, then we'll stop there. You might agree that many modern Kerberos features are just not available yet.
>
> In my view we have to tweak it for ourselves, that's why I spent so much time on it. The framework originates the one from MIT Kerberos and is very flexible, say once a new encryption type is up then it's easy to add it accordingly so we are updated! The support is complete and all the ones provided in MIT Kerberos are done. I'm confident with it, basically I would think it's the core part that's essential for a Kerberos implementation. Without it, maybe it's fine for us to just replace the Kerberos codec part based on existing Kerberos implementation. Yes we must be careful with the codes. I would be very happy for your review of it and make it strong enough. 

Ok, seems like I have to do my homework in this area ;-)


>>> has also to be considered is the configuration part, and integration with Studio.
> Yes agree. Regarding Kerberos configuration either in client side (krb5.conf) or server side (kdc.conf), I guess we'd better make it compatible with MIT Kerberos so that administrators won't have to learn about it. Also in our documentation for simple we would just say "please reference MIT Krb5 doc" as we're going the same way. A contributor is work on it and will commit the codes once the codebase is imported.
That's not that simple. Of course, we should support MIT based
configuration, but you also have to consider those admin who have to
install a kerberos server and don't know about tjose files : they most
certainly expect the product we deliver to take care of the config
without having to deal with .conf files...

RE: About Haox project

["Zheng, Kai" <ka...@intel.com>]

Hi Emmanuel,

Thanks for the history and background. Very interesting and helpful for me to understand ApacheDS project and the community. I would like to contribute and develop in either codes or community, particularly from the point of view interested and inspired by Hadoop ecosystem.

>> Thanks Kai. At this point, I think we could launch a vote to accept the contribution.
Thanks for holding this. Meanwhile, I welcome any review and feedback for the project codes, if any issue I should resolve first I will. Currently I'm working on the cleanup and clearance to prepare for this.

>> I must admit that the ASN/1 codec you are coming from is not only way simpler than what we use, but it's also faster !
Thanks for your help tuning it even faster! With the help of the community experts, I will be much confident about making the codec edible. Security field (Kerberos, PKI, LDAP and etc.) are very ASN1 heavy, and a simpler one would help for new contributors. 

>> I wanted to build a ASN/1 compiler (see http://svn.apache.org/repos/asf/labs/dungeon/)
>> Implementing a new Codec requires a huge effort, even for me.
It's very interesting about the compiler. Actually it was in my mind to support such compiler for haox-asn1 library, enabling to compile an ASN1 definition module (like krb5.asn1) and automatically generate all the necessary Java types (like Kerberos codec classes), and we don't even bother to code them manually.  Yes it's hard, huge effort is involved, not easy for me either.

>> I would rather first check what Bouncy Castle has to propose in this area. I'd be really careful here, implementing some crypto amgorithm is seriously complex, and require a lot of care.
Sorry I'm confusing and have to be clear here. When I mentioned "implement all the encryption types and checksum types that MIT Kerberos supports, like DES, DES3, AES, RC4, CAMELLIA", I mean we need to implement the Kerberos encryption types and checksum types which are often mentioned as Kerberos crypto. Such crypto isn't the one in JCE neither any in BC, where only covers concrete encryption algorithms. Kerberos crypto is based on concrete encryption algorithms and implementations, but would never bother to define and implement any encryption algorithms itself.

Yes we could rely on the Kerberos crypto implemented in JRE, as existing ApacheDS Kerberos implementation does. The problems are: 1) JRE doesn't expose the functionalities fully and is very limited; 2) It's not complete and we can't guarantee full interoperable with MIT Kerberos and MS-AD; 3) Relying on JRE means we'll never be able to go beyond that, if the corresponding support isn't available in JRE, then we'll stop there. You might agree that many modern Kerberos features are just not available yet.

In my view we have to tweak it for ourselves, that's why I spent so much time on it. The framework originates the one from MIT Kerberos and is very flexible, say once a new encryption type is up then it's easy to add it accordingly so we are updated! The support is complete and all the ones provided in MIT Kerberos are done. I'm confident with it, basically I would think it's the core part that's essential for a Kerberos implementation. Without it, maybe it's fine for us to just replace the Kerberos codec part based on existing Kerberos implementation. Yes we must be careful with the codes. I would be very happy for your review of it and make it strong enough. 

>> The base idea is still to be able to deliver a global package where the Kerberos Server is accessing ApacheDS LDAP Server without going through the network
Yes, I agree. With such an abstraction and API defined, various identity back ends, either internal embedded or external crossing network can be supported. One thing to note is concrete identity backend implementation would be separate from the Kerberos Server in its own pom-project so never enforces any dependencies. In runtime, such a backend will be dynamically searched or loaded from configuration then employed. 

>> has also to be considered is the configuration part, and integration with Studio.
Yes agree. Regarding Kerberos configuration either in client side (krb5.conf) or server side (kdc.conf), I guess we'd better make it compatible with MIT Kerberos so that administrators won't have to learn about it. Also in our documentation for simple we would just say "please reference MIT Krb5 doc" as we're going the same way. A contributor is work on it and will commit the codes once the codebase is imported.

>> we want to have something that move forward, not just to replace what we curently have
Yes totally agree. That's why we're here. Develop the project, develop the community, making the implementation updated and strong enough as a Kerberos distribution.

>> This is a critical part of the success !
Yes, with such possible inspiring by "Apache Kerberos", we would attract and welcome more contributors in future. I guess when we come to the 2nd phase, we would be able to update to the Hadoop space and it's possible we'll get more contribution and feedback from there. Currently Hadoop becomes one of the biggest platforms that deploy Kerberos, and various challenges are met there. Hopefully we can help them further. Note ApacheDS Kerberos was already integrated for unit tests.

>> That sounds like a perfect proposal.
Thank you!

Regards,
Kai

-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com] 
Sent: Friday, December 26, 2014 4:39 PM
To: dev@directory.apache.org
Subject: Re: About Haox project

Le 23/12/14 18:57, Zheng, Kai a écrit :
> Hi all,

Hi Kai,

sorry for the delayed mail, I had to digest my Xmas heavy dinner and lunch ;-)
>
> I'm Kai and a guy from Intel Big Data team. I have been working on Apache Hadoop for some while, and used to be an ApacheDS contributor and committed the feature of CredentialCahce and a simple Kinit tool, guided by Emmanuel and Kiran. After that when I was going to implement more Kerberos features I realized there needs important improvement space in the existing fundamental Kerberos codes.
You bet...

>  As a prototype I initiated Haox project  and have implemented an ASN1 encoding/decoding framework, complete encryption and checksum crypto, Kerberos codec, Keytab and CredentialCache facilities, KrbClient API and simple KDC server. It's still in its initial phase and far from ideal. However as strongly suggested by Emmanuel and Kiran, I should collaborate with ApacheDS community and continue to make the effort with you together. It's great and now I'm going to contribute the project to ApacheDS. I have discussed with Emmanuel and Kiran for quite some time about this. Based on our discussion, we have the following proposal. It's for your review and consider. Please kindly provide your feedback and advise. Thanks in advance.

Thanks Kai. At this point, I think we could launch a vote to accept the contribution.
>
> 1. First all our vision for the long term, we would think it makes great sense to establish a first class Kerberos implementation in Java for the Apache world targeting today's environments  in cloud, Hadoop and mobile. Therefore we could collaborate together in ApacheDS community to build a "Apache Kerberos" project. That's why we're here.

This was one of the reason we developped the Kerberos Server (well, sort of). Let's say the history of our current implementation is a bit complex. When Alex started to work on the LDAP server, he designed a server that would be embeddable, and which could process more than just the LDAP protocol. Kerberos was a natural choce - as was DNS, DHCP, etc... Basically, the idea was to have what we call a DirectoryService which handles the data, and a front-end to process the incoming requests (the Protocol layer). The current kerberos implementation is based on this DirectoryService and is tightly coupled with the protocol layer.
This effort, sadly, staled for years before we rewrote most of the ASN.1 layer - which was far from being perfect - and added a Kerberos client plus fixed many issues. But, still, we have so many things on pur plate that it's hard for us to move forward fast enough.

What would your proposal change ? Well, After havinbg looked at it, I do think it's what we should expect from a Kerberos server : it should not necessarily being tied to a LDAP server as it is in our current code, and I must admit that the ASN/1 codec you are coming from is not only way simpler than what we use, but it's also faster !

A digression here : the very first thing I worked on back in 2005 when I jumped into the project was a rewrite of the existing ASN/1 codec, which was slow like death (well, it was not exactly the very first thing, as I get accepted for my small contrinution on the LDIF parser, but anyway).
Due to the constraints (ie, the codec had to be statefull, capable of dealing with TCP fragmentation) I came with a slution based on a State Machine. It was also driven by the fact I wanted to build a ASN/1 compiler (see http://svn.apache.org/repos/asf/labs/dungeon/), a staled effort. The codec is efficient, up to a point, but extremelly complex to use. Implementing a new Codec requires a huge effort, even for me.


>
> 2. And as the first step to start with, ApacheDS would accept Haox as 
> a sub-project, renamed as "Apache Kerberos". Then we consolidate 
> existing Kerberos implementation codes in ApacheDS into the new 
> project;

+1
>
> 3. Afterwards we will make further effort to push "Apache Kerberos" out as a top level project, probably in a year or so.

That's a decision which will be easy, as soon as we have enough contributor to the project.
>
> We have discussed other approaches, like using the good stuffs in Haox to replace the corresponding parts in ApacheDS Kerberos, as one may also suggest. So let's figure out what might be good parts implemented in Haox.
>
> 1.     Haox-ASN1. It's a schema/model driven ASN1 parser and codec. Based on the framework new Kerberos types can be easily defined following fixed patterns, without having to take care of ASN1 encoding/decoding procedures. We see this as very important because Kerberos itself and related extensions are very ASN1 heavy. Without the basic friendly library support it's hard for new developers to contribute on new Kerberos evolving features. We tested the codec performance and it outperforms a little. Emmanuel has even contributed a few optimization improvements.
+1
>
>
>
> 2.     Kerberos crypto. The crypto framework (based on RFC3961) and library implement all the encryption types and checksum types that MIT Kerberos supports, like DES, DES3, AES, RC4, CAMELLIA, with extensive interoperable tests covered. It relies on JCE for encryption but not any Kerberos facilities provided in JRE. Potentially we could port it to other non-Oracle Java environment like Android system, assuming we can implement corresponding encryption and hash providers which can be pluggable.

I would rather first check what Bouncy Castle has to propose in this area. I'd be really careful here, implementing some crypto amgorithm is seriously complex, and require a lot of care.

>
>
> 3.     Kerberos codec. Based on Haox-Asn1, the codec has implemented all the necessary Kerberos types defined in Krb5(RFC4210), Preauth/FAST(RFC6113), and PKINIT(4556). The core part has some test cases covered already.

We should just migrate the tests we have in our implementation to test the new one. Easy, even if might be boring.
>
>
> 4.     KDC server. The implementation abstracts an KrbIdentityService API and decouples any specific identity backend from the Kerberos server. We can provide an LDAP based backend, or allow vendor to implement their own backed by SQL database, external LDAP server or whatever. For simple unit test usage, a memory HashMap is all the needs. This allows our embedded KDC really very lightweight avoiding many unnecessary dependencies.

A decoupling must be done in ApacheDS. That should not be a big deal.
The base idea is still to be able to deliver a global package where the Kerberos Server is accessing ApacheDS LDAP Server without going through the network, ie by using the CoreSession API, or may be defining a higher API which uses CoreSession under the hood : this higher level API would connect to a LDAP server (whatever it is) through this API, and we would either go through the network, or uses CoreSession.
>
>
> 5.     Haox-event. It implements a UDP/TCP mixed network event support based on Java NIO and reactor pattern according to Kerberos's needs, where UDP and TCP are both required, and even in IAKERB, a KDC proxy would serve both TCP server role and UDP/TCP client role. In Haox-event, all these roles can be done in an unified event receiving, processing and replying. Test cases are given but it definitely need to tune and improve. Again, it doesn't rely on any other libraries, thus based on it KrbClient could easily be supported in other Java environment.

Here, I would check if MINA could be a good substitue, or not. It's still an open question.
>
> Keeping our long term goal in mind, we thought above functionalities, properties and considerations may serve better to establish the foundation for "Apache Kerberos" project, and it would be much easier to consolidate the existing Kerberos implementation into it.

What has also to be considered is the configuration part, and integration with Studio.

>
> So as a first step for the collaboration, how about:
>
> 1.     Prepare, which involves:
>
> 1) Necessary cleanup;
>
> 2) All Kerberos related codes will start package names with "org.apache.kerberos", all the non-Kerberos related codes will be moved into contrib projects or libraries, like Haox-ASN1, Haox-event, Haox-config, which are actually not Kerberos logic specific.

We should think about that, sure.
>
> 2. Move Haox to be an ApacheDS sub-project:
> 1) Renaming Haox as "Apache Kerberos";
> 2) A separate GIT repo;
> 3) A separate coding style, subject to be discussed and determined.
+1
>
> Then as the 2nd step, consolidate the necessary parts in existing Kerberos implementation. Below is the initial list and please complement:
>
> 1.     Apache Directory Server support. We need to create an LdapIdentityBackend using Apache Directory Server and use it by default, embedded in the KDC server.
>
> 2.     Installation packaging and service wrapper. Haox hasn't implemented such but ApacheDS does.
>
> 3.     Tests. All the Kerberos related tests should be evaluated and ported to the new codebase if not obsolete.

+1
>
> In addition to above, we also need to consider the following aspects, to make it good enough for potential users.
>
> 1.     Adding the missing Javadoc.
>
> 2.     Adding more tests.
>
> 3.     Adding the documentation.
> 4. Building the web site

+1. Part of the process anyway.
>
> With such done, then we might obsolete the AacheDS Kerberos implementation and might move all the related codes into an history folder for existing users. We would encourage users to use the new project.
> For the 3rd step, we would continue to implement some essential and advanced features, which may include:
>
> 1.     Cross realm support, which means Apache Kerberos can federate with MIT Kerberos or MS AD;
>
> 2.     Full functional client tools, like kinit, klist and kadmin, or other alternatives;
>
> 3.     Preauth/FAST framework, which allows to support other authentication mechanisms based on the secure channel protected by an Armor TGT/key;
>
> 4.     PKINIT, authenticating user using x509 certificate other than plain password;
>
> 5.     Token preauth, authenticating user using a JWT token, to support OAuth.
>
> 6.     ...

All in all, that are the important points : we want to have something that move forward, not just to replace what we curently have, which is somehow not moving forward at a decent pace (and here, I'm ebing quite kind to the current satte...)
>
> Note tasks in step2 and 3 are not necessarily to be done in the order. We're likely to work on them in parallel.
>
> We would encourage more developers to contribute to the effort and develop the community. Without that, we may not be able to make it to have a high quality of Kerberos implementation, with all the important Kerberos feature gaps being filled.

This is a critical part of the success !
>
> Then finally as the last step, in some appropriate time we would push Apache Kerberos out to be an Apache top level project.
>
> All the points are just suggestions and subject to your discussion. Your feedback is most welcome. As widely discussed in a week or so, this proposal is to be finalized and then re-posted. Then let's collaborate and make it together.

That sounds like a perfect proposal.

Thanks Kai !

Re: About Haox project

[Emmanuel Lécharny <el...@gmail.com>]

Le 23/12/14 18:57, Zheng, Kai a écrit :
> Hi all,

Hi Kai,

sorry for the delayed mail, I had to digest my Xmas heavy dinner and
lunch ;-)
>
> I'm Kai and a guy from Intel Big Data team. I have been working on Apache Hadoop for some while, and used to be an ApacheDS contributor and committed the feature of CredentialCahce and a simple Kinit tool, guided by Emmanuel and Kiran. After that when I was going to implement more Kerberos features I realized there needs important improvement space in the existing fundamental Kerberos codes.
You bet...

>  As a prototype I initiated Haox project  and have implemented an ASN1 encoding/decoding framework, complete encryption and checksum crypto, Kerberos codec, Keytab and CredentialCache facilities, KrbClient API and simple KDC server. It's still in its initial phase and far from ideal. However as strongly suggested by Emmanuel and Kiran, I should collaborate with ApacheDS community and continue to make the effort with you together. It's great and now I'm going to contribute the project to ApacheDS. I have discussed with Emmanuel and Kiran for quite some time about this. Based on our discussion, we have the following proposal. It's for your review and consider. Please kindly provide your feedback and advise. Thanks in advance.

Thanks Kai. At this point, I think we could launch a vote to accept the
contribution.
>
> 1. First all our vision for the long term, we would think it makes great sense to establish a first class Kerberos implementation in Java for the Apache world targeting today's environments  in cloud, Hadoop and mobile. Therefore we could collaborate together in ApacheDS community to build a "Apache Kerberos" project. That's why we're here.

This was one of the reason we developped the Kerberos Server (well, sort
of). Let's say the history of our current implementation is a bit
complex. When Alex started to work on the LDAP server, he designed a
server that would be embeddable, and which could process more than just
the LDAP protocol. Kerberos was a natural choce - as was DNS, DHCP,
etc... Basically, the idea was to have what we call a DirectoryService
which handles the data, and a front-end to process the incoming requests
(the Protocol layer). The current kerberos implementation is based on
this DirectoryService and is tightly coupled with the protocol layer.
This effort, sadly, staled for years before we rewrote most of the ASN.1
layer - which was far from being perfect - and added a Kerberos client
plus fixed many issues. But, still, we have so many things on pur plate
that it's hard for us to move forward fast enough.

What would your proposal change ? Well, After havinbg looked at it, I do
think it's what we should expect from a Kerberos server : it should not
necessarily being tied to a LDAP server as it is in our current code,
and I must admit that the ASN/1 codec you are coming from is not only
way simpler than what we use, but it's also faster !

A digression here : the very first thing I worked on back in 2005 when I
jumped into the project was a rewrite of the existing ASN/1 codec, which
was slow like death (well, it was not exactly the very first thing, as I
get accepted for my small contrinution on the LDIF parser, but anyway).
Due to the constraints (ie, the codec had to be statefull, capable of
dealing with TCP fragmentation) I came with a slution based on a State
Machine. It was also driven by the fact I wanted to build a ASN/1
compiler (see http://svn.apache.org/repos/asf/labs/dungeon/), a staled
effort. The codec is efficient, up to a point, but extremelly complex to
use. Implementing a new Codec requires a huge effort, even for me.


>
> 2. And as the first step to start with, ApacheDS would accept Haox as a sub-project, renamed as "Apache Kerberos". Then we consolidate existing Kerberos implementation codes in ApacheDS into the new project;

+1
>
> 3. Afterwards we will make further effort to push "Apache Kerberos" out as a top level project, probably in a year or so.

That's a decision which will be easy, as soon as we have enough
contributor to the project.
>
> We have discussed other approaches, like using the good stuffs in Haox to replace the corresponding parts in ApacheDS Kerberos, as one may also suggest. So let's figure out what might be good parts implemented in Haox.
>
> 1.     Haox-ASN1. It's a schema/model driven ASN1 parser and codec. Based on the framework new Kerberos types can be easily defined following fixed patterns, without having to take care of ASN1 encoding/decoding procedures. We see this as very important because Kerberos itself and related extensions are very ASN1 heavy. Without the basic friendly library support it's hard for new developers to contribute on new Kerberos evolving features. We tested the codec performance and it outperforms a little. Emmanuel has even contributed a few optimization improvements.
+1
>
>
>
> 2.     Kerberos crypto. The crypto framework (based on RFC3961) and library implement all the encryption types and checksum types that MIT Kerberos supports, like DES, DES3, AES, RC4, CAMELLIA, with extensive interoperable tests covered. It relies on JCE for encryption but not any Kerberos facilities provided in JRE. Potentially we could port it to other non-Oracle Java environment like Android system, assuming we can implement corresponding encryption and hash providers which can be pluggable.

I would rather first check what Bouncy Castle has to propose in this
area. I'd be really careful here, implementing some crypto amgorithm is
seriously complex, and require a lot of care.

>
>
> 3.     Kerberos codec. Based on Haox-Asn1, the codec has implemented all the necessary Kerberos types defined in Krb5(RFC4210), Preauth/FAST(RFC6113), and PKINIT(4556). The core part has some test cases covered already.

We should just migrate the tests we have in our implementation to test
the new one. Easy, even if might be boring.
>
>
> 4.     KDC server. The implementation abstracts an KrbIdentityService API and decouples any specific identity backend from the Kerberos server. We can provide an LDAP based backend, or allow vendor to implement their own backed by SQL database, external LDAP server or whatever. For simple unit test usage, a memory HashMap is all the needs. This allows our embedded KDC really very lightweight avoiding many unnecessary dependencies.

A decoupling must be done in ApacheDS. That should not be a big deal.
The base idea is still to be able to deliver a global package where the
Kerberos Server is accessing ApacheDS LDAP Server without going through
the network, ie by using the CoreSession API, or may be defining a
higher API which uses CoreSession under the hood : this higher level API
would connect to a LDAP server (whatever it is) through this API, and we
would either go through the network, or uses CoreSession.
>
>
> 5.     Haox-event. It implements a UDP/TCP mixed network event support based on Java NIO and reactor pattern according to Kerberos's needs, where UDP and TCP are both required, and even in IAKERB, a KDC proxy would serve both TCP server role and UDP/TCP client role. In Haox-event, all these roles can be done in an unified event receiving, processing and replying. Test cases are given but it definitely need to tune and improve. Again, it doesn't rely on any other libraries, thus based on it KrbClient could easily be supported in other Java environment.

Here, I would check if MINA could be a good substitue, or not. It's
still an open question.
>
> Keeping our long term goal in mind, we thought above functionalities, properties and considerations may serve better to establish the foundation for "Apache Kerberos" project, and it would be much easier to consolidate the existing Kerberos implementation into it.

What has also to be considered is the configuration part, and
integration with Studio.

>
> So as a first step for the collaboration, how about:
>
> 1.     Prepare, which involves:
>
> 1) Necessary cleanup;
>
> 2) All Kerberos related codes will start package names with "org.apache.kerberos", all the non-Kerberos related codes will be moved into contrib projects or libraries, like Haox-ASN1, Haox-event, Haox-config, which are actually not Kerberos logic specific.

We should think about that, sure.
>
> 2. Move Haox to be an ApacheDS sub-project:
> 1) Renaming Haox as "Apache Kerberos";
> 2) A separate GIT repo;
> 3) A separate coding style, subject to be discussed and determined.
+1
>
> Then as the 2nd step, consolidate the necessary parts in existing Kerberos implementation. Below is the initial list and please complement:
>
> 1.     Apache Directory Server support. We need to create an LdapIdentityBackend using Apache Directory Server and use it by default, embedded in the KDC server.
>
> 2.     Installation packaging and service wrapper. Haox hasn't implemented such but ApacheDS does.
>
> 3.     Tests. All the Kerberos related tests should be evaluated and ported to the new codebase if not obsolete.

+1
>
> In addition to above, we also need to consider the following aspects, to make it good enough for potential users.
>
> 1.     Adding the missing Javadoc.
>
> 2.     Adding more tests.
>
> 3.     Adding the documentation.
> 4. Building the web site

+1. Part of the process anyway.
>
> With such done, then we might obsolete the AacheDS Kerberos implementation and might move all the related codes into an history folder for existing users. We would encourage users to use the new project.
> For the 3rd step, we would continue to implement some essential and advanced features, which may include:
>
> 1.     Cross realm support, which means Apache Kerberos can federate with MIT Kerberos or MS AD;
>
> 2.     Full functional client tools, like kinit, klist and kadmin, or other alternatives;
>
> 3.     Preauth/FAST framework, which allows to support other authentication mechanisms based on the secure channel protected by an Armor TGT/key;
>
> 4.     PKINIT, authenticating user using x509 certificate other than plain password;
>
> 5.     Token preauth, authenticating user using a JWT token, to support OAuth.
>
> 6.     ...

All in all, that are the important points : we want to have something
that move forward, not just to replace what we curently have, which is
somehow not moving forward at a decent pace (and here, I'm ebing quite
kind to the current satte...)
>
> Note tasks in step2 and 3 are not necessarily to be done in the order. We're likely to work on them in parallel.
>
> We would encourage more developers to contribute to the effort and develop the community. Without that, we may not be able to make it to have a high quality of Kerberos implementation, with all the important Kerberos feature gaps being filled.

This is a critical part of the success !
>
> Then finally as the last step, in some appropriate time we would push Apache Kerberos out to be an Apache top level project.
>
> All the points are just suggestions and subject to your discussion. Your feedback is most welcome. As widely discussed in a week or so, this proposal is to be finalized and then re-posted. Then let's collaborate and make it together.

That sounds like a perfect proposal.

Thanks Kai !