UTF-8 password encoding issue with Java 1.6 releases

[dan <da...@bamlabs.com>]

Hi --

We are having difficulty because the UTF-8 encoding has changed between Java
1.6 releases.  In particular, we are running 1.6.0_10 and find that
passwords are encoded to UTF-8 format differently than in 1.6.0_24.  Here is
some official documentation describing it:

http://www.oracle.com/technetwork/java/javase/6u11-139394.html

My understanding is that Shiro relies on the UTF-8 format being "standard"
and reproducible.  Interestingly, Sun also made a change to the UTF-8 format
in Java 1.5.

In Shiro, the password is entered as a String and then converted to UTF-8
format before hashing.  As you can imagine, I'm having a lot of trouble
trying to figure out how to migrate old passwords to a new format!!  But, I
wanted to make you aware of this issue -- perhaps, Shiro should encode using
UTF-32 format instead since that format truly can't change.

Dan


--
View this message in context: http://shiro-user.582556.n2.nabble.com/UTF-8-password-encoding-issue-with-Java-1-6-releases-tp6308011p6308011.html
Sent from the Shiro User mailing list archive at Nabble.com.

Re: UTF-8 password encoding issue with Java 1.6 releases

[dan <da...@bamlabs.com>]

Hi Les,

This issue arose for me because of some weird utf8 encoding/decoding
transformation that was done in our pre-Shiro authentication implementation. 
I solved it this backwards compatibility issue by including the utf8
encoder/decoder from openjdk in my project and calling it for old passwords.

Anyway, it did get me thinking about UTF8... For "normal" passwords, such as
those in low-order ascii or common languages, I would expect the encoding to
be stable between java versions.  It looks like there are some corner cases
where a character encoding might change, so if someone specifies a password
with one of these oddball cases, password matching will fail between Java
versions.  It seems safer to use UTF32 for all plain password to byte array
conversions since it is a simpler transformation than UTF8.  But that said,
UTF8 seems like a reasonable implementation for 99.999% of use cases.

Best,
Dan

--
View this message in context: http://shiro-user.582556.n2.nabble.com/UTF-8-password-encoding-issue-with-Java-1-6-releases-tp6308011p6314348.html
Sent from the Shiro User mailing list archive at Nabble.com.

Re: UTF-8 password encoding issue with Java 1.6 releases

[Les Hazlewood <lh...@apache.org>]

Thanks for the heads-up Dan.

This is truly bizarre - I can't see how UTF-8, being a 'standard',
could ever change.  It was just not something designed in Shiro ever
change.  I mean, who is to say that Sun's implementation of UTF-32 is
not susceptible to the same problems now?  What a blunder by Sun....

There is a potential solution around this that I can think of - we
have a codec package that has a CodecSupport class used elsewhere in
the project.  If someone would contribute the 'old' UTF-8
implementation to Shiro, we can make that a pluggable strategy into
our codec stuff.  The default would probably be the JVM default, but
you could configure whatever you prefer.  Please feel free to open a
Jira issue to address this if you think it is worth solving.

Best,

Les

On Tue, Apr 26, 2011 at 7:54 PM, dan <da...@bamlabs.com> wrote:
> Hi --
>
> We are having difficulty because the UTF-8 encoding has changed between Java
> 1.6 releases.  In particular, we are running 1.6.0_10 and find that
> passwords are encoded to UTF-8 format differently than in 1.6.0_24.  Here is
> some official documentation describing it:
>
> http://www.oracle.com/technetwork/java/javase/6u11-139394.html
>
> My understanding is that Shiro relies on the UTF-8 format being "standard"
> and reproducible.  Interestingly, Sun also made a change to the UTF-8 format
> in Java 1.5.
>
> In Shiro, the password is entered as a String and then converted to UTF-8
> format before hashing.  As you can imagine, I'm having a lot of trouble
> trying to figure out how to migrate old passwords to a new format!!  But, I
> wanted to make you aware of this issue -- perhaps, Shiro should encode using
> UTF-32 format instead since that format truly can't change.
>
> Dan
>
>
> --
> View this message in context: http://shiro-user.582556.n2.nabble.com/UTF-8-password-encoding-issue-with-Java-1-6-releases-tp6308011p6308011.html
> Sent from the Shiro User mailing list archive at Nabble.com.