You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@santuario.apache.org by Colm O hEigeartaigh <co...@apache.org> on 2016/05/30 10:30:24 UTC
Base64 error?
Hi,
I'm doing some testing with various BASE-64 implementations and I think
there's an error in the Santuario Base64 encoder to do with whitespace. If
so though it's been there a looong time without anyone noticing...
The BASE64 implementation is here:
https://svn.apache.org/repos/asf/santuario/xml-security-java/trunk/src/main/java/org/apache/xml/security/utils/Base64.java
In the "encode" method it's adding a newline character with:
encodedData[encodedIndex++] = 0xa;
However this is just "\n". The RFC defines a CRLF as "\r\n":
https://www.ietf.org/rfc/rfc2045.txt
It looks like a bug...but would like some feedback from others more
familiar with the RFC.
Colm.
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: Base64 error?
Posted by "Cantor, Scott" <ca...@osu.edu>.
> I did some interop testing with the Commons Codec Base64 implementation
> and the JDK8 java.util one, and the output is different. I have to explicitly use
> new byte[] {'\n'} for the line break to get them to work with Santuario.
Could you define "work with"?
I know we use commons in our code and I don't think we've had issues consuming signatures. If you mean "get them to produce the same encoded output", that's understandable.
-- Scott
Re: Base64 error?
Posted by Colm O hEigeartaigh <co...@apache.org>.
Fixed here: http://svn.apache.org/viewvc?view=revision&revision=1747188
Colm.
On Tue, May 31, 2016 at 3:02 PM, Cantor, Scott <ca...@osu.edu> wrote:
> > XML Schema requires processors to ignore the line-length limitation and
> > collapse whitespace sequences to single space characters, so in schema
> > terms this all becomes irrelevant, and XML Sig says the schema is
> > normative. But it also says MIME is normative. Which is more normative
> > than the other?
>
> XML Schema is not a presumed part of the XML Signature processing model,
> and in fact XML Signature fails badly in a number of scenarios involving
> schema validation, so it is generally bad practice to combine them without
> a lot of extra care.
>
> -- Scott
>
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: Base64 error?
Posted by "Cantor, Scott" <ca...@osu.edu>.
> XML Schema requires processors to ignore the line-length limitation and
> collapse whitespace sequences to single space characters, so in schema
> terms this all becomes irrelevant, and XML Sig says the schema is
> normative. But it also says MIME is normative. Which is more normative
> than the other?
XML Schema is not a presumed part of the XML Signature processing model, and in fact XML Signature fails badly in a number of scenarios involving schema validation, so it is generally bad practice to combine them without a lot of extra care.
-- Scott
Re: Base64 error?
Posted by Dennis Sosnoski <dm...@sosnoski.com>.
I'd thought that the line length restriction had been overridden
somewhere for XML Sig, but I don't see where that's specified. There was
a discussion of this whole issue with the intent to change it:
https://lists.w3.org/Archives/Public/w3c-ietf-xmldsig/2001AprJun/0195.html
XML Schema requires processors to ignore the line-length limitation and
collapse whitespace sequences to single space characters, so in schema
terms this all becomes irrelevant, and XML Sig says the schema is
normative. But it also says MIME is normative. Which is more normative
than the other?
- Dennis
On 05/31/2016 05:49 AM, Marc Giger wrote:
> Hi Colm,
>
> XMLDsig and XMLEnc referring RFC 2045 (Base64 MIME) as you
> already noted. After reading many sources I also come to
> the conclusion that
>
> a) Line breaks MUST be inserted after 76 characters
> b) Line breaks MUST be CRLF sequences
>
> The Java-Doc of java.util.Base64 and
> org.apache.commons.codec.binary.Base64
> also confirms that statement.
>
> To get the same output from both implementations you
> can/should use
> java.util.Base64.getMimeEncoder() // RFC2045
> and
> org.apache.commons.codec.binary.Base64.Base64(final boolean
> urlSafe) // RFC2045
>
> or other constructors while specifying explicitly the line lengths and
> newline characters.
>
>
> In the case of commons codec the _decoder_ should work with none/LF/CRLF
> because it simply ignores unexpected values.
> I can imagine that other decoder impl. do the same. Santuario
> should most probably do the same for max. interoperability.
>
> Btw, it looks like I got it also wrong (because I didn't read the RFC)
> in the StAX impl. If it happens that you fix the Base64 impl. could
> you also correct the new Base64OutputStream(...) call in
> AbstractEncryptOutputProcessor ?
>
> Thanks and happy coding,
>
> Marc
>
>
>
>
> On Mon, 30 May 2016 14:32:30 +0100
> Colm O hEigeartaigh <co...@apache.org> wrote:
>
>> I did some interop testing with the Commons Codec Base64
>> implementation and the JDK8 java.util one, and the output is
>> different. I have to explicitly use new byte[] {'\n'} for the line
>> break to get them to work with Santuario.
>>
>> Colm.
>>
>> On Mon, May 30, 2016 at 1:49 PM, Alessio Soldano <as...@redhat.com>
>> wrote:
>>
>>> I wonder which implications this could have in terms of
>>> interoperability... ?
>>>
>>>
>>> Il 30/05/2016 12:30, Colm O hEigeartaigh ha scritto:
>>>
>>>> Hi,
>>>>
>>>> I'm doing some testing with various BASE-64 implementations and I
>>>> think there's an error in the Santuario Base64 encoder to do with
>>>> whitespace. If so though it's been there a looong time without
>>>> anyone noticing...
>>>>
>>>> The BASE64 implementation is here:
>>>>
>>>>
>>>> https://svn.apache.org/repos/asf/santuario/xml-security-java/trunk/src/main/java/org/apache/xml/security/utils/Base64.java
>>>>
>>>> In the "encode" method it's adding a newline character with:
>>>>
>>>> encodedData[encodedIndex++] = 0xa;
>>>>
>>>> However this is just "\n". The RFC defines a CRLF as "\r\n":
>>>>
>>>> https://www.ietf.org/rfc/rfc2045.txt
>>>>
>>>> It looks like a bug...but would like some feedback from others more
>>>> familiar with the RFC.
>>>>
>>>> Colm.
>>>>
>>>>
>>>> --
>>>> Colm O hEigeartaigh
>>>>
>>>> Talend Community Coder
>>>> http://coders.talend.com
>>>>
>>>
>>> --
>>> Alessio Soldano
>>> Web Service Lead, JBoss
>>>
>>>
>>
>> --
>> Colm O hEigeartaigh
>>
>> Talend Community Coder
>> http://coders.talend.com
>
Re: Base64 error?
Posted by Marc Giger <gi...@apache.org>.
Hi Colm,
XMLDsig and XMLEnc referring RFC 2045 (Base64 MIME) as you
already noted. After reading many sources I also come to
the conclusion that
a) Line breaks MUST be inserted after 76 characters
b) Line breaks MUST be CRLF sequences
The Java-Doc of java.util.Base64 and
org.apache.commons.codec.binary.Base64
also confirms that statement.
To get the same output from both implementations you
can/should use
java.util.Base64.getMimeEncoder() // RFC2045
and
org.apache.commons.codec.binary.Base64.Base64(final boolean
urlSafe) // RFC2045
or other constructors while specifying explicitly the line lengths and
newline characters.
In the case of commons codec the _decoder_ should work with none/LF/CRLF
because it simply ignores unexpected values.
I can imagine that other decoder impl. do the same. Santuario
should most probably do the same for max. interoperability.
Btw, it looks like I got it also wrong (because I didn't read the RFC)
in the StAX impl. If it happens that you fix the Base64 impl. could
you also correct the new Base64OutputStream(...) call in
AbstractEncryptOutputProcessor ?
Thanks and happy coding,
Marc
On Mon, 30 May 2016 14:32:30 +0100
Colm O hEigeartaigh <co...@apache.org> wrote:
> I did some interop testing with the Commons Codec Base64
> implementation and the JDK8 java.util one, and the output is
> different. I have to explicitly use new byte[] {'\n'} for the line
> break to get them to work with Santuario.
>
> Colm.
>
> On Mon, May 30, 2016 at 1:49 PM, Alessio Soldano <as...@redhat.com>
> wrote:
>
> > I wonder which implications this could have in terms of
> > interoperability... ?
> >
> >
> > Il 30/05/2016 12:30, Colm O hEigeartaigh ha scritto:
> >
> >> Hi,
> >>
> >> I'm doing some testing with various BASE-64 implementations and I
> >> think there's an error in the Santuario Base64 encoder to do with
> >> whitespace. If so though it's been there a looong time without
> >> anyone noticing...
> >>
> >> The BASE64 implementation is here:
> >>
> >>
> >> https://svn.apache.org/repos/asf/santuario/xml-security-java/trunk/src/main/java/org/apache/xml/security/utils/Base64.java
> >>
> >> In the "encode" method it's adding a newline character with:
> >>
> >> encodedData[encodedIndex++] = 0xa;
> >>
> >> However this is just "\n". The RFC defines a CRLF as "\r\n":
> >>
> >> https://www.ietf.org/rfc/rfc2045.txt
> >>
> >> It looks like a bug...but would like some feedback from others more
> >> familiar with the RFC.
> >>
> >> Colm.
> >>
> >>
> >> --
> >> Colm O hEigeartaigh
> >>
> >> Talend Community Coder
> >> http://coders.talend.com
> >>
> >
> >
> > --
> > Alessio Soldano
> > Web Service Lead, JBoss
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
Re: Base64 error?
Posted by Colm O hEigeartaigh <co...@apache.org>.
I did some interop testing with the Commons Codec Base64 implementation and
the JDK8 java.util one, and the output is different. I have to explicitly
use new byte[] {'\n'} for the line break to get them to work with Santuario.
Colm.
On Mon, May 30, 2016 at 1:49 PM, Alessio Soldano <as...@redhat.com>
wrote:
> I wonder which implications this could have in terms of
> interoperability... ?
>
>
> Il 30/05/2016 12:30, Colm O hEigeartaigh ha scritto:
>
>> Hi,
>>
>> I'm doing some testing with various BASE-64 implementations and I think
>> there's an error in the Santuario Base64 encoder to do with whitespace. If
>> so though it's been there a looong time without anyone noticing...
>>
>> The BASE64 implementation is here:
>>
>>
>> https://svn.apache.org/repos/asf/santuario/xml-security-java/trunk/src/main/java/org/apache/xml/security/utils/Base64.java
>>
>> In the "encode" method it's adding a newline character with:
>>
>> encodedData[encodedIndex++] = 0xa;
>>
>> However this is just "\n". The RFC defines a CRLF as "\r\n":
>>
>> https://www.ietf.org/rfc/rfc2045.txt
>>
>> It looks like a bug...but would like some feedback from others more
>> familiar with the RFC.
>>
>> Colm.
>>
>>
>> --
>> Colm O hEigeartaigh
>>
>> Talend Community Coder
>> http://coders.talend.com
>>
>
>
> --
> Alessio Soldano
> Web Service Lead, JBoss
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: Base64 error?
Posted by Alessio Soldano <as...@redhat.com>.
I wonder which implications this could have in terms of
interoperability... ?
Il 30/05/2016 12:30, Colm O hEigeartaigh ha scritto:
> Hi,
>
> I'm doing some testing with various BASE-64 implementations and I
> think there's an error in the Santuario Base64 encoder to do with
> whitespace. If so though it's been there a looong time without anyone
> noticing...
>
> The BASE64 implementation is here:
>
> https://svn.apache.org/repos/asf/santuario/xml-security-java/trunk/src/main/java/org/apache/xml/security/utils/Base64.java
>
> In the "encode" method it's adding a newline character with:
>
> encodedData[encodedIndex++] = 0xa;
>
> However this is just "\n". The RFC defines a CRLF as "\r\n":
>
> https://www.ietf.org/rfc/rfc2045.txt
>
> It looks like a bug...but would like some feedback from others more
> familiar with the RFC.
>
> Colm.
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
--
Alessio Soldano
Web Service Lead, JBoss
Re: Base64 error?
Posted by Colm O hEigeartaigh <co...@apache.org>.
Thanks all for the feedback. I came across this bug in Apache Ranger, which
uses the Santuario BASE-64 encoder to encode keys as part of a Hadoop Key
Management Service. I swapped the Santuario BASE-64 encoder out and
replaced it with Commons Codec, and the output is different. To produce the
same output, I need to do something like:
String masterKey1 =
org.apache.xml.security.utils.Base64.encode(key.getBytes());
org.apache.commons.codec.binary.Base64 base64 = new
org.apache.commons.codec.binary.Base64(76, new byte[]{'\n'});
String masterKey2 = new String(base64.encode(key.getBytes())).trim();
org.junit.Assert.assertEquals(masterKey1, masterKey2);
There is actually two problems here, using '\n' instead of '\r\n' and also
not adding an additional line separator at the end (hence the trim() above
on the commons codec output).
Given, as Clemont points out, that the XML processor maps \r\n to \n on
input, I'm inclined to make the change as it won't have huge backwards
compatibility implications. Anyone object to this?
Colm.
On Mon, May 30, 2016 at 9:09 PM, Cantor, Scott <ca...@osu.edu> wrote:
> > The XML spec requires the XML processor to map \r\n to \n on input,
> > thereby hiding this discrepancy.
>
> Good point.
>
> > Is the Base64 value ever used without this mapping taking place?
>
> Possibly in very obscure cases involving enveloping signatures?
>
> -- Scott
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
RE: Base64 error?
Posted by "Cantor, Scott" <ca...@osu.edu>.
> The XML spec requires the XML processor to map \r\n to \n on input,
> thereby hiding this discrepancy.
Good point.
> Is the Base64 value ever used without this mapping taking place?
Possibly in very obscure cases involving enveloping signatures?
-- Scott
RE: Base64 error?
Posted by "Pellerin, Clement" <Cl...@ibi.com>.
> However this is just "\n". The RFC defines a CRLF as "\r\n":
The XML spec requires the XML processor to map \r\n to \n on input,
thereby hiding this discrepancy.
Is the Base64 value ever used without this mapping taking place?
RE: Base64 error?
Posted by "Cantor, Scott" <ca...@osu.edu>.
> However this is just "\n". The RFC defines a CRLF as "\r\n":
>
> https://www.ietf.org/rfc/rfc2045.txt
>
> It looks like a bug...but would like some feedback from others more familiar
> with the RFC.
The RFC seems pretty clear, and unlike some specs, XML Sig is pretty explicit about referencing it.
The C++ library delegates base64 to the cryptographic libraries, and I don't know offhand what OpenSSL does by default.
Most base64 decoders are very liberal about accepting anything re: whitespace, obviously. But if you're going to fix this, I would allow some time for people to test it.
-- Scott