You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Oliver Deakin <ol...@googlemail.com> on 2010/08/02 12:08:36 UTC
Re: [classlib][x-net] Creating a provider based on OpenSSL
I've made a little progress with this in my local workspace, but I'd
like to get the work out into the community so everyone can see it. To
this end I've created a temporary branch [1] under which I will be
committing changes as I go. I figured this was the easiest thing so I
can reorganise things, break the build etc. without disturbing our main
tree.
So far I've got the following:
- I'm passing some of the relevant certs/keys down to native code and
then into OpenSSL. I'm not 100% sure what I've done with this is right
so far, but it's work in progress!
- OpenSSL is performing the SSL handshake and, at least for a simple
client-server case, is succeeding with the certs passed from Java.
- I've modified the read/write methods on SSLSocketImpl to make calls
through natives into OpenSSL and they seem to be working. It's a little
bit hacked at the moment because I'm working from the starting point of
the JSSE implementation we already have, but the fact it works at least
verifies that the connection has been successfully made and OpenSSL has
been configured correctly.
I'll be committing these changes to the sandbox I've just created soon.
Please feel free to check out the tree and take a look/submit patches if
you like :) Things I still need to investigate are:
- Is the method I've used to pass down certs to OpenSSL sufficient?
- Reads from the socket are 1 byte at a time at the moment(!) - this
needs to be altered to use the OpenSSL calls more efficiently.
- How do I tell OpenSSL to use the Java SecureRandom implementation
rather than its own random number generator? Via an OpenSSL ENGINE?
- What's the best way to get/build the OpenSSL dependency?
Any comments/suggestions are welcome!
Regards,
Oliver
[1] https://svn.apache.org/repos/asf/harmony/enhanced/java/branches/omd
On 19/07/2010 17:15, Oliver Deakin wrote:
> Hi all,
>
> I'm currently investigating the possibility of implementing a JSSE
> provider wrapping OpenSSL. This has a couple of obvious advantages:
> - The onus of code maintenance and bug fixing in a security sensitive
> area is moved outside of Harmony.
> - New protocols can be integrated into the Harmony provider with
> minimal effort (updating dependencies rather than implementing them
> ourselves).
>
> Really I'm sending this mail as a heads up, but would be interested to
> know if anyone has any experience/opinions in this area. In
> particular, I'd be interested in ideas on:
> - the best way to setup OpenSSL as a dependency - precompile the
> libraries and make them available for download or compile them at
> build time on the user's machine.
> - how to tie in the Java x-net APIs to the OpenSSL APIs.
>
> Any comments/suggestions welcome.
>
> Regards,
> Oliver
>
--
Oliver Deakin
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [classlib][x-net] Creating a provider based on OpenSSL
Posted by Oliver Deakin <ol...@googlemail.com>.
Thanks Cath, that will be a great help! I've only been testing on
Windows so far, so it will be very useful if you can check how things
are looking on Linux as we progress.
Thanks for the heads up on the Java classes - I've just rebuilt and see
the build failures too now. I think the problem was that I was
incrementally building. In the Java build this doesn't necessarily mean
that the whole dependency tree is built when a class is changed, so you
can break dependencies other classes have without noticing. I have some
changes I plan to commit soon that should fix these problems.
Regards,
Oliver
On 18/08/2010 09:29, Catherine Hope wrote:
> Hi Oli,
>
> I can help you work on this. I've created a JIRA
> https://issues.apache.org/jira/browse/HARMONY-6627 to attach patches to. So
> far I've had to make a few changes to build on linux and currently the Java
> classes don't build from scratch.
>
> Best wishes,
> Cath
>
--
Oliver Deakin
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [classlib][x-net] Creating a provider based on OpenSSL
Posted by Catherine Hope <ca...@googlemail.com>.
Hi Oli,
I can help you work on this. I've created a JIRA
https://issues.apache.org/jira/browse/HARMONY-6627 to attach patches to. So
far I've had to make a few changes to build on linux and currently the Java
classes don't build from scratch.
Best wishes,
Cath
Re: [classlib][x-net] Creating a provider based on OpenSSL
Posted by Oliver Deakin <ol...@googlemail.com>.
On 20/08/2010 20:59, Oliver Deakin wrote:
> On 20/08/2010 18:26, Catherine Hope wrote:
>> I've been working on SSLSocketInputStream, which currently reads
>> bytes one
>> at a time. I've attached a patch to
>> https://issues.apache.org/jira/browse/HARMONY-6627 with an
>> implementation of
>> the read methods using OpenSSL calls and replaced the available call
>> to use
>> ioctl.
>
> Thanks for the new patch Cath - I applied it with some modifications,
> comments added to the JIRA.
>
>> Currently when I run the x-net tests I have 2 failures and 11
>> errors. With
>> my fix the org.apache.harmony.xnet.provider.jsse.SSLSocketFunctionalTest
>> gets a bit further - it's currently failing with a
>> javax.net.ssl.SSLHandshakeException:
>> no shared cipher
>
> I see the same failures. For the "no shared cipher" the cause is not
> immediately obvious - from the list of available ciphers returned by
> OpenSSL I would expect the ciphers used in this test to all succeed.
> I'll add some debug to both sides of the connection and see if that
> tells me anything.
Fixed at revision r988108. The test now passes completely for me. Please
let me know if you still see failures.
Regards,
Oliver
>
>> I've encountered some problems using the keytool to get some examples to
>> work - these are raised under
>> https://issues.apache.org/jira/browse/HARMONY-6631 with a fix.
>
> I think this would be a good issue to get fixed before the next
> milestone - I'll try and get this applied before the code freeze.
>
> Regards,
> Oliver
>
>
>
--
Oliver Deakin
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [classlib][x-net] Creating a provider based on OpenSSL
Posted by Oliver Deakin <ol...@googlemail.com>.
On 20/08/2010 18:26, Catherine Hope wrote:
> I've been working on SSLSocketInputStream, which currently reads bytes one
> at a time. I've attached a patch to
> https://issues.apache.org/jira/browse/HARMONY-6627 with an implementation of
> the read methods using OpenSSL calls and replaced the available call to use
> ioctl.
Thanks for the new patch Cath - I applied it with some modifications,
comments added to the JIRA.
> Currently when I run the x-net tests I have 2 failures and 11 errors. With
> my fix the org.apache.harmony.xnet.provider.jsse.SSLSocketFunctionalTest
> gets a bit further - it's currently failing with a
> javax.net.ssl.SSLHandshakeException:
> no shared cipher
I see the same failures. For the "no shared cipher" the cause is not
immediately obvious - from the list of available ciphers returned by
OpenSSL I would expect the ciphers used in this test to all succeed.
I'll add some debug to both sides of the connection and see if that
tells me anything.
> I've encountered some problems using the keytool to get some examples to
> work - these are raised under
> https://issues.apache.org/jira/browse/HARMONY-6631 with a fix.
I think this would be a good issue to get fixed before the next
milestone - I'll try and get this applied before the code freeze.
Regards,
Oliver
--
Oliver Deakin
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [classlib][x-net] Creating a provider based on OpenSSL
Posted by Catherine Hope <ca...@googlemail.com>.
I've been working on SSLSocketInputStream, which currently reads bytes one
at a time. I've attached a patch to
https://issues.apache.org/jira/browse/HARMONY-6627 with an implementation of
the read methods using OpenSSL calls and replaced the available call to use
ioctl.
Currently when I run the x-net tests I have 2 failures and 11 errors. With
my fix the org.apache.harmony.xnet.provider.jsse.SSLSocketFunctionalTest
gets a bit further - it's currently failing with a
javax.net.ssl.SSLHandshakeException:
no shared cipher
I've encountered some problems using the keytool to get some examples to
work - these are raised under
https://issues.apache.org/jira/browse/HARMONY-6631 with a fix.
Re: [classlib][x-net] Creating a provider based on OpenSSL
Posted by Oliver Deakin <ol...@googlemail.com>.
Hi Chris,
On 19/08/2010 21:40, chris.gray@k-embedded-java.com wrote:
> Hi Oliver,
>
>>> - Set a single global set of native RNG callback functions that then
>>> figures out (somehow) who made the original call into the OpenSSL
>>> natives and uses the appropriate SecureRandom implementation. This
>>> would be tricky, but might be possible. I imagine we could look up the
>>> stack for the last Java (i.e. non-native) frame and could get the
>>> SecureRandom reference from that class.
>> I went for something like this option in the end using ThreadLocals to
>> get back my SSLParameters class when we're in the RNG functions (thanks
>> to Cath Hope for this idea!). I'm not completely satisfied with the
>> solution, but it works and is fairly simple. In the case where no
>> SecureRandom implementation is provided the code falls back to the
>> OpenSSL default RNG functions.
> The only problem I see with this is that it assumes that all calls to a
> particular SSL engine instance will be made from a single thread.
I think the solution I've implemented should work in multiple threads as
well. It follows these steps:
- when a thread is about to make a native call that will call OpenSSL
it sets a thread local with its SSLParameters instance. This means that
even if multiple threads share the same SSL engine and SSLParameters
they will each set their own thread local containing the correct
SecureRandom instance to use.
- the natives do whatever they need to. If the RNG functions are
called they lookup the thread local set in the first step and use the
SecureRandom implementation if it is set. If no SecureRandom has been
set, the code falls back to calling the OpenSSL default RNG.
- when the natives have returned, the thread removes its thread local
and continues.
This should mean that each thread has its own thread local correctly set
whenever we are in OpenSSL calls (at least those calls that might use
RNG functions).
> That may
> not be the case if for example the SSL connection is established by a
> "main" or "startup" thread and then messages will be sent on that
> connection by an EventListener which is run by a quite different thread.
> That would lead to extremely puzzling bugs 8-0. From that point of view I
> would prefer an implementation which simply ignored the user-supplied
> SecureRandom (and clearly documented this behaviour).
Yep, and I agree in that case, but I think the solution I have should
work in multiple threads so hopefully we can satisfy the spec.
> I also have to say that if this is the biggest unsolved problem you still
> have, I'm pretty impressed.
It may be that I just haven't discovered the biggest problem yet ;)
Regards,
Oliver
> Best regards
>
> Chris Gray
>
>
>
>
>
>
>
--
Oliver Deakin
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [classlib][x-net] Creating a provider based on OpenSSL
Posted by ch...@k-embedded-java.com.
Hi Oliver,
>> - Set a single global set of native RNG callback functions that then
>> figures out (somehow) who made the original call into the OpenSSL
>> natives and uses the appropriate SecureRandom implementation. This
>> would be tricky, but might be possible. I imagine we could look up the
>> stack for the last Java (i.e. non-native) frame and could get the
>> SecureRandom reference from that class.
>
> I went for something like this option in the end using ThreadLocals to
> get back my SSLParameters class when we're in the RNG functions (thanks
> to Cath Hope for this idea!). I'm not completely satisfied with the
> solution, but it works and is fairly simple. In the case where no
> SecureRandom implementation is provided the code falls back to the
> OpenSSL default RNG functions.
The only problem I see with this is that it assumes that all calls to a
particular SSL engine instance will be made from a single thread. That may
not be the case if for example the SSL connection is established by a
"main" or "startup" thread and then messages will be sent on that
connection by an EventListener which is run by a quite different thread.
That would lead to extremely puzzling bugs 8-0. From that point of view I
would prefer an implementation which simply ignored the user-supplied
SecureRandom (and clearly documented this behaviour).
I also have to say that if this is the biggest unsolved problem you still
have, I'm pretty impressed.
Best regards
Chris Gray
Re: [classlib][x-net] Creating a provider based on OpenSSL
Posted by Oliver Deakin <ol...@googlemail.com>.
On 16/08/2010 13:47, Oliver Deakin wrote:
> I've been making steady progress with this implementation so far but
> I've hit a snag. The Java spec allows users to specify a SecureRandom
> implementation when creating connections, and there can be different
> SecureRandom implementations for different connections in the same VM
> instance. However, OpenSSL only allows one global random number
> generator (RNG) to be specified. This means that currently there is no
> way to fully satisfy the Java spec. I see 3 options for implementing
> this:
>
> - Implement the additional APIs to do this in OpenSSL ourselves or
> ask the OpenSSL project to do it. I'm not sure if there is much need
> for this from OpenSSL's point of view so I don't think this is a very
> good option.
> - Set a single global set of native RNG callback functions that then
> figures out (somehow) who made the original call into the OpenSSL
> natives and uses the appropriate SecureRandom implementation. This
> would be tricky, but might be possible. I imagine we could look up the
> stack for the last Java (i.e. non-native) frame and could get the
> SecureRandom reference from that class.
I went for something like this option in the end using ThreadLocals to
get back my SSLParameters class when we're in the RNG functions (thanks
to Cath Hope for this idea!). I'm not completely satisfied with the
solution, but it works and is fairly simple. In the case where no
SecureRandom implementation is provided the code falls back to the
OpenSSL default RNG functions.
Regards,
Oliver
> - We just ignore the SecureRandom class passed in by the user or only
> use one of the specified implementations. This would be a departure
> from the spec but in practise should not make any difference to the
> security of the connection. I think it would be unlikely that users
> would specify multiple different SecureRandom implementations (or
> specify any at all in most cases), so this may be a reasonable solution.
>
> I'm going to investigate the 2nd option as I think this offers us a
> way to satisfy the spec without modifying OpenSSL and allowing us to
> work with previous versions also.
>
> Any thoughts/comments?
>
> Regards,
> Oliver
>
> On 02/08/2010 11:08, Oliver Deakin wrote:
>> <SNIP>
>>
>> Regards,
>> Oliver
>>
>> [1] https://svn.apache.org/repos/asf/harmony/enhanced/java/branches/omd
>>
>> On 19/07/2010 17:15, Oliver Deakin wrote:
>>> Hi all,
>>>
>>> I'm currently investigating the possibility of implementing a JSSE
>>> provider wrapping OpenSSL. This has a couple of obvious advantages:
>>> - The onus of code maintenance and bug fixing in a security
>>> sensitive area is moved outside of Harmony.
>>> - New protocols can be integrated into the Harmony provider with
>>> minimal effort (updating dependencies rather than implementing them
>>> ourselves).
>>>
>>> Really I'm sending this mail as a heads up, but would be interested
>>> to know if anyone has any experience/opinions in this area. In
>>> particular, I'd be interested in ideas on:
>>> - the best way to setup OpenSSL as a dependency - precompile the
>>> libraries and make them available for download or compile them at
>>> build time on the user's machine.
>>> - how to tie in the Java x-net APIs to the OpenSSL APIs.
>>>
>>> Any comments/suggestions welcome.
>>>
>>> Regards,
>>> Oliver
>>>
>>
>
--
Oliver Deakin
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [classlib][x-net] Creating a provider based on OpenSSL
Posted by Oliver Deakin <ol...@googlemail.com>.
I've been making steady progress with this implementation so far but
I've hit a snag. The Java spec allows users to specify a SecureRandom
implementation when creating connections, and there can be different
SecureRandom implementations for different connections in the same VM
instance. However, OpenSSL only allows one global random number
generator (RNG) to be specified. This means that currently there is no
way to fully satisfy the Java spec. I see 3 options for implementing this:
- Implement the additional APIs to do this in OpenSSL ourselves or ask
the OpenSSL project to do it. I'm not sure if there is much need for
this from OpenSSL's point of view so I don't think this is a very good
option.
- Set a single global set of native RNG callback functions that then
figures out (somehow) who made the original call into the OpenSSL
natives and uses the appropriate SecureRandom implementation. This would
be tricky, but might be possible. I imagine we could look up the stack
for the last Java (i.e. non-native) frame and could get the SecureRandom
reference from that class.
- We just ignore the SecureRandom class passed in by the user or only
use one of the specified implementations. This would be a departure from
the spec but in practise should not make any difference to the security
of the connection. I think it would be unlikely that users would specify
multiple different SecureRandom implementations (or specify any at all
in most cases), so this may be a reasonable solution.
I'm going to investigate the 2nd option as I think this offers us a way
to satisfy the spec without modifying OpenSSL and allowing us to work
with previous versions also.
Any thoughts/comments?
Regards,
Oliver
On 02/08/2010 11:08, Oliver Deakin wrote:
> <SNIP>
>
> Regards,
> Oliver
>
> [1] https://svn.apache.org/repos/asf/harmony/enhanced/java/branches/omd
>
> On 19/07/2010 17:15, Oliver Deakin wrote:
>> Hi all,
>>
>> I'm currently investigating the possibility of implementing a JSSE
>> provider wrapping OpenSSL. This has a couple of obvious advantages:
>> - The onus of code maintenance and bug fixing in a security
>> sensitive area is moved outside of Harmony.
>> - New protocols can be integrated into the Harmony provider with
>> minimal effort (updating dependencies rather than implementing them
>> ourselves).
>>
>> Really I'm sending this mail as a heads up, but would be interested
>> to know if anyone has any experience/opinions in this area. In
>> particular, I'd be interested in ideas on:
>> - the best way to setup OpenSSL as a dependency - precompile the
>> libraries and make them available for download or compile them at
>> build time on the user's machine.
>> - how to tie in the Java x-net APIs to the OpenSSL APIs.
>>
>> Any comments/suggestions welcome.
>>
>> Regards,
>> Oliver
>>
>
--
Oliver Deakin
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU