You are viewing a plain text version of this content. The canonical link for it is here.
Posted to xmlrpc-dev@ws.apache.org by Dustin Sallings <du...@spy.net> on 2002/06/27 19:27:36 UTC
base64 interop problems
Hi, I'm using the Apache XML-RPC code in a project and I had some
bad interoperability problems when using a non-Apache client.
Specifically, when I used the python client to send a binary of
about 700k, it gained about 10k in the decoding. This seems to be because
the decoder does not do anything to skip input characters that are not
valid input characters. In particular, section 6.8 of RFC 2045 states the
following:
The encoded output stream must be represented in lines of no more
than 76 characters each. All line breaks or other characters not
found in Table 1 must be ignored by decoding software. In base64
data, characters other than those in Table 1, line breaks, and other
white space probably indicate a transmission error, about which a
warning message or even a message rejection might be appropriate
under some circumstances.
The python encoder was indeed sending 76 character lines, which
seemed to be the problem.
Not wanting to spend a lot of time getting this thing going, I
replaced the base64 decoder with one that I had written and have done
interop testing with. The patch is included, and you guys are free to do
what you want with it.
Like the original decoder, it does not throw an exception on
invalid input, it instead tried to decode as best as it can. It may be
more appropriate here to throw an exception if the invalid character is
not a whitespace, i.e. change this:
// Skip over invalid characters.
if(x<0) {
invalid++;
continue;
}
to
// Skip over invalid characters.
if(x<0) {
if(!Character.isWhitespace(input.charAt(i))) {
throw new IOException("Invalid Base64 character found");
}
// Otherwise, mark it and continue
invalid++;
continue;
}
Anyway, you guys are free to do with this code as you please, but
I'm using it in the meantime to make my app work.
--
SPY My girlfriend asked me which one I like better.
pub 1024/3CAE01D5 1994/11/03 Dustin Sallings <du...@spy.net>
| Key fingerprint = 87 02 57 08 02 D0 DA D6 C8 0F 3E 65 51 98 D8 BE
L_______________________ I hope the answer won't upset her. ____________
Re: base64 interop problems
Posted by Dustin Sallings <du...@spy.net>.
Around 14:04 on Sep 28, 2002, John Wilson said:
# Did you look at the performance of your new version? The existing code
# is not as fast as it could be. It would be very nice if you have
# improved performance as well as fixing a bug :-)
I didn't do a comparison, but I'm sure it's not too bad. I bet I
could find some places to speed it up if I tried, just need a test
framework.
--
SPY My girlfriend asked me which one I like better.
pub 1024/3CAE01D5 1994/11/03 Dustin Sallings <du...@spy.net>
| Key fingerprint = 87 02 57 08 02 D0 DA D6 C8 0F 3E 65 51 98 D8 BE
L_______________________ I hope the answer won't upset her. ____________
Re: base64 interop problems
Posted by Dustin Sallings <du...@spy.net>.
Around 14:04 on Sep 28, 2002, John Wilson said:
# Did you look at the performance of your new version? The existing code
# is not as fast as it could be. It would be very nice if you have
# improved performance as well as fixing a bug :-)
I didn't do a comparison, but I'm sure it's not too bad. I bet I
could find some places to speed it up if I tried, just need a test
framework.
--
SPY My girlfriend asked me which one I like better.
pub 1024/3CAE01D5 1994/11/03 Dustin Sallings <du...@spy.net>
| Key fingerprint = 87 02 57 08 02 D0 DA D6 C8 0F 3E 65 51 98 D8 BE
L_______________________ I hope the answer won't upset her. ____________
Re: base64 interop problems
Posted by John Wilson <tu...@wilson.co.uk>.
----- Original Message -----
From: "Dustin Sallings" <du...@spy.net>
To: "Daniel Rall" <dl...@finemaltcoding.com>
Cc: <rp...@xml.apache.org>
Sent: Saturday, September 28, 2002 12:56 AM
Subject: Re: base64 interop problems
[snip]
> The issue is the handling of non-base64 data in the input of the
> decoder. The base64 decoder attempts to decode every character in the
> input stream. The one I provided skips characters that are not base64
> characters. A third behavior is probably more appropriate: skip only
> whitespace characters and throw an exception on non-whitespace, non-base64
> characters.
Yes I think that it is better to throw an exception on non whitespace.
Did you look at the performance of your new version? The existing code is
not as fast as it could be. It would be very nice if you have improved
performance as well as fixing a bug :-)
John Wilson
The Wilson Partnership
http://www.wilson.co.uk
Re: base64 interop problems
Posted by John Wilson <tu...@wilson.co.uk>.
----- Original Message -----
From: "Dustin Sallings" <du...@spy.net>
To: "Daniel Rall" <dl...@finemaltcoding.com>
Cc: <rp...@xml.apache.org>
Sent: Saturday, September 28, 2002 12:56 AM
Subject: Re: base64 interop problems
[snip]
> The issue is the handling of non-base64 data in the input of the
> decoder. The base64 decoder attempts to decode every character in the
> input stream. The one I provided skips characters that are not base64
> characters. A third behavior is probably more appropriate: skip only
> whitespace characters and throw an exception on non-whitespace, non-base64
> characters.
Yes I think that it is better to throw an exception on non whitespace.
Did you look at the performance of your new version? The existing code is
not as fast as it could be. It would be very nice if you have improved
performance as well as fixing a bug :-)
John Wilson
The Wilson Partnership
http://www.wilson.co.uk
Re: base64 interop problems
Posted by Dustin Sallings <du...@spy.net>.
Around 16:42 on Sep 27, 2002, Daniel Rall said:
# Dustin, would you comment on that issue listed above? Your patch came
# through as changing the entire Base64.java file, rather than just a
# small section of code.
Yeah, I just needed to get it running, so I didn't try to fix what
was there (well, I did, but I didn't spend a lot of time in it). Instead,
I replaced it with one I'd written that I knew works.
I ran into the problem when adding XML-RPC support to an
application I've written (in java) that moves photographs around, and then
creating a standalone app (in python) to talk to it. The data size was
not the same upon decoding as it was before sending it up. With the
changes I've made, it seems to work very well. I now have clients written
in python (xmlrpclib), java (using this framework), perl (I didn't write
the perl one, but it works), and objective C (using a framework built upon
xmlrpc-c) all working with the application now.
The issue is the handling of non-base64 data in the input of the
decoder. The base64 decoder attempts to decode every character in the
input stream. The one I provided skips characters that are not base64
characters. A third behavior is probably more appropriate: skip only
whitespace characters and throw an exception on non-whitespace, non-base64
characters.
I hope that's clear, it's Friday. :)
--
SPY My girlfriend asked me which one I like better.
pub 1024/3CAE01D5 1994/11/03 Dustin Sallings <du...@spy.net>
| Key fingerprint = 87 02 57 08 02 D0 DA D6 C8 0F 3E 65 51 98 D8 BE
L_______________________ I hope the answer won't upset her. ____________
Re: base64 interop problems
Posted by Dustin Sallings <du...@spy.net>.
Around 16:42 on Sep 27, 2002, Daniel Rall said:
# Dustin, would you comment on that issue listed above? Your patch came
# through as changing the entire Base64.java file, rather than just a
# small section of code.
Yeah, I just needed to get it running, so I didn't try to fix what
was there (well, I did, but I didn't spend a lot of time in it). Instead,
I replaced it with one I'd written that I knew works.
I ran into the problem when adding XML-RPC support to an
application I've written (in java) that moves photographs around, and then
creating a standalone app (in python) to talk to it. The data size was
not the same upon decoding as it was before sending it up. With the
changes I've made, it seems to work very well. I now have clients written
in python (xmlrpclib), java (using this framework), perl (I didn't write
the perl one, but it works), and objective C (using a framework built upon
xmlrpc-c) all working with the application now.
The issue is the handling of non-base64 data in the input of the
decoder. The base64 decoder attempts to decode every character in the
input stream. The one I provided skips characters that are not base64
characters. A third behavior is probably more appropriate: skip only
whitespace characters and throw an exception on non-whitespace, non-base64
characters.
I hope that's clear, it's Friday. :)
--
SPY My girlfriend asked me which one I like better.
pub 1024/3CAE01D5 1994/11/03 Dustin Sallings <du...@spy.net>
| Key fingerprint = 87 02 57 08 02 D0 DA D6 C8 0F 3E 65 51 98 D8 BE
L_______________________ I hope the answer won't upset her. ____________
Re: base64 interop problems
Posted by Daniel Rall <dl...@finemaltcoding.com>.
Dustin Sallings <du...@spy.net> writes:
> Around 14:39 on Jul 1, 2002, Daniel Rall said:
>
> # Hey Dustin. Is this be related to issue 9931, "Base64 decoder chokes on
> # a whitespace?"
> #
> # http://issues.apache.org/bugzilla/show_bug.cgi?id=9931
>
> Well, that'd be exactly my problem. :)
>
> Sorry for not investigating further. My goal was to make my
> application work, and after doing so, I thought someone else would
> probably have had the same problem I did. I looked around for list
> archives, but didn't think to look for a bug tracking system.
>
> Anyway, the one I sent in also does not throw an exception, and
> may ``succesfully'' decode something that got corrupt and contains invalid
> characters as it ignores any invalid characters (easy to fix, as stated in
> my original mail), but the behavior does seem more correct. At least, my
> application is working between java and python now.
Dustin, would you comment on that issue listed above? Your patch came
through as changing the entire Base64.java file, rather than just a
small section of code.
Thanks, Dan
--
Daniel Rall <dl...@finemaltcoding.com>
Re: base64 interop problems
Posted by Daniel Rall <dl...@finemaltcoding.com>.
Dustin Sallings <du...@spy.net> writes:
> Around 14:39 on Jul 1, 2002, Daniel Rall said:
>
> # Hey Dustin. Is this be related to issue 9931, "Base64 decoder chokes on
> # a whitespace?"
> #
> # http://issues.apache.org/bugzilla/show_bug.cgi?id=9931
>
> Well, that'd be exactly my problem. :)
>
> Sorry for not investigating further. My goal was to make my
> application work, and after doing so, I thought someone else would
> probably have had the same problem I did. I looked around for list
> archives, but didn't think to look for a bug tracking system.
>
> Anyway, the one I sent in also does not throw an exception, and
> may ``succesfully'' decode something that got corrupt and contains invalid
> characters as it ignores any invalid characters (easy to fix, as stated in
> my original mail), but the behavior does seem more correct. At least, my
> application is working between java and python now.
Dustin, would you comment on that issue listed above? Your patch came
through as changing the entire Base64.java file, rather than just a
small section of code.
Thanks, Dan
--
Daniel Rall <dl...@finemaltcoding.com>
Re: base64 interop problems
Posted by Dustin Sallings <du...@spy.net>.
Around 14:39 on Jul 1, 2002, Daniel Rall said:
# Hey Dustin. Is this be related to issue 9931, "Base64 decoder chokes on
# a whitespace?"
#
# http://issues.apache.org/bugzilla/show_bug.cgi?id=9931
Well, that'd be exactly my problem. :)
Sorry for not investigating further. My goal was to make my
application work, and after doing so, I thought someone else would
probably have had the same problem I did. I looked around for list
archives, but didn't think to look for a bug tracking system.
Anyway, the one I sent in also does not throw an exception, and
may ``succesfully'' decode something that got corrupt and contains invalid
characters as it ignores any invalid characters (easy to fix, as stated in
my original mail), but the behavior does seem more correct. At least, my
application is working between java and python now.
--
SPY My girlfriend asked me which one I like better.
pub 1024/3CAE01D5 1994/11/03 Dustin Sallings <du...@spy.net>
| Key fingerprint = 87 02 57 08 02 D0 DA D6 C8 0F 3E 65 51 98 D8 BE
L_______________________ I hope the answer won't upset her. ____________
Re: base64 interop problems
Posted by Dustin Sallings <du...@spy.net>.
Around 14:39 on Jul 1, 2002, Daniel Rall said:
# Hey Dustin. Is this be related to issue 9931, "Base64 decoder chokes on
# a whitespace?"
#
# http://issues.apache.org/bugzilla/show_bug.cgi?id=9931
Well, that'd be exactly my problem. :)
Sorry for not investigating further. My goal was to make my
application work, and after doing so, I thought someone else would
probably have had the same problem I did. I looked around for list
archives, but didn't think to look for a bug tracking system.
Anyway, the one I sent in also does not throw an exception, and
may ``succesfully'' decode something that got corrupt and contains invalid
characters as it ignores any invalid characters (easy to fix, as stated in
my original mail), but the behavior does seem more correct. At least, my
application is working between java and python now.
--
SPY My girlfriend asked me which one I like better.
pub 1024/3CAE01D5 1994/11/03 Dustin Sallings <du...@spy.net>
| Key fingerprint = 87 02 57 08 02 D0 DA D6 C8 0F 3E 65 51 98 D8 BE
L_______________________ I hope the answer won't upset her. ____________
Re: base64 interop problems
Posted by Daniel Rall <dl...@finemaltcoding.com>.
Hey Dustin. Is this be related to issue 9931, "Base64 decoder chokes
on a whitespace?"
http://issues.apache.org/bugzilla/show_bug.cgi?id=9931
Dustin Sallings <du...@spy.net> writes:
> Hi, I'm using the Apache XML-RPC code in a project and I had some
> bad interoperability problems when using a non-Apache client.
>
> Specifically, when I used the python client to send a binary of
> about 700k, it gained about 10k in the decoding. This seems to be because
> the decoder does not do anything to skip input characters that are not
> valid input characters. In particular, section 6.8 of RFC 2045 states the
> following:
>
> The encoded output stream must be represented in lines of no more
> than 76 characters each. All line breaks or other characters not
> found in Table 1 must be ignored by decoding software. In base64
> data, characters other than those in Table 1, line breaks, and other
> white space probably indicate a transmission error, about which a
> warning message or even a message rejection might be appropriate
> under some circumstances.
>
> The python encoder was indeed sending 76 character lines, which
> seemed to be the problem.
>
> Not wanting to spend a lot of time getting this thing going, I
> replaced the base64 decoder with one that I had written and have done
> interop testing with. The patch is included, and you guys are free to do
> what you want with it.
>
> Like the original decoder, it does not throw an exception on
> invalid input, it instead tried to decode as best as it can. It may be
> more appropriate here to throw an exception if the invalid character is
> not a whitespace, i.e. change this:
>
> // Skip over invalid characters.
> if(x<0) {
> invalid++;
> continue;
> }
>
> to
>
> // Skip over invalid characters.
> if(x<0) {
> if(!Character.isWhitespace(input.charAt(i))) {
> throw new IOException("Invalid Base64 character found");
> }
> // Otherwise, mark it and continue
> invalid++;
> continue;
> }
>
> Anyway, you guys are free to do with this code as you please, but
> I'm using it in the meantime to make my app work.
>
> --
> SPY My girlfriend asked me which one I like better.
> pub 1024/3CAE01D5 1994/11/03 Dustin Sallings <du...@spy.net>
> | Key fingerprint = 87 02 57 08 02 D0 DA D6 C8 0F 3E 65 51 98 D8 BE
> L_______________________ I hope the answer won't upset her. ____________
Re: base64 interop problems
Posted by Daniel Rall <dl...@finemaltcoding.com>.
Hey Dustin. Is this be related to issue 9931, "Base64 decoder chokes
on a whitespace?"
http://issues.apache.org/bugzilla/show_bug.cgi?id=9931
Dustin Sallings <du...@spy.net> writes:
> Hi, I'm using the Apache XML-RPC code in a project and I had some
> bad interoperability problems when using a non-Apache client.
>
> Specifically, when I used the python client to send a binary of
> about 700k, it gained about 10k in the decoding. This seems to be because
> the decoder does not do anything to skip input characters that are not
> valid input characters. In particular, section 6.8 of RFC 2045 states the
> following:
>
> The encoded output stream must be represented in lines of no more
> than 76 characters each. All line breaks or other characters not
> found in Table 1 must be ignored by decoding software. In base64
> data, characters other than those in Table 1, line breaks, and other
> white space probably indicate a transmission error, about which a
> warning message or even a message rejection might be appropriate
> under some circumstances.
>
> The python encoder was indeed sending 76 character lines, which
> seemed to be the problem.
>
> Not wanting to spend a lot of time getting this thing going, I
> replaced the base64 decoder with one that I had written and have done
> interop testing with. The patch is included, and you guys are free to do
> what you want with it.
>
> Like the original decoder, it does not throw an exception on
> invalid input, it instead tried to decode as best as it can. It may be
> more appropriate here to throw an exception if the invalid character is
> not a whitespace, i.e. change this:
>
> // Skip over invalid characters.
> if(x<0) {
> invalid++;
> continue;
> }
>
> to
>
> // Skip over invalid characters.
> if(x<0) {
> if(!Character.isWhitespace(input.charAt(i))) {
> throw new IOException("Invalid Base64 character found");
> }
> // Otherwise, mark it and continue
> invalid++;
> continue;
> }
>
> Anyway, you guys are free to do with this code as you please, but
> I'm using it in the meantime to make my app work.
>
> --
> SPY My girlfriend asked me which one I like better.
> pub 1024/3CAE01D5 1994/11/03 Dustin Sallings <du...@spy.net>
> | Key fingerprint = 87 02 57 08 02 D0 DA D6 C8 0F 3E 65 51 98 D8 BE
> L_______________________ I hope the answer won't upset her. ____________