You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Julius Davies <ju...@gmail.com> on 2008/05/31 09:06:44 UTC
[codec] Base64InputStream
Hi,
I know of a good Base64InputStream for commons-codec. It can encode
and decode. It uses the Base64 class from commons-codec under the
hood. It works like this:
-------------------------------------------
final boolean DECODE = true;
final boolean ENCODE = false;
InputStream in = new FileInputStream("file.base64");
in = new Base64InputStream(in, DECODE);
-------------------------------------------
I ran some performance tests (decode 699MB file) against some other
Base64 implementations:
1. [60.507 seconds] http://iharder.sourceforge.net/base64/
2. [69.311 seconds] sun.misc.BASE64Decoder
3. [31.018 seconds] "openssl enc -base64 -d" (non-java unix command)
4. [30.865 seconds] not-yet-commons-ssl THE WINNER!!!!
- Core 2 Duo 2.8ghz on Mac Leopard
- Sun's 1.5.0_13 with "-server"
- Full 700MB was decoded three times in a single go (no JVM restarts).
The shortest time for each is printed above.
- Files created by the 4 techniques compared using md5sum. All look same.
- BufferedInputStream and BufferedOutputStream wrappers.
- (If I may brag: not-yet-commons-ssl performance doesn't change if
BufferedInputStream / BufferedOutputStream aren't used. The other
libraries really suffer - like 20 times slower!)
I also tried to test with ws-common, but the Base64 library in there
doesn't have a streaming decoder, so I don't think it can handle a
700MB file.
Here's the code:
http://juliusdavies.ca/svn/viewvc.cgi/not-yet-commons-ssl/trunk/src/java/org/apache/commons/ssl/Base64InputStream.java?view=markup
I should mention one thing.... a huge IO bottleneck was fixed in
not-yet-commons-ssl by doing this:
byte[] line = readLine.nextAsBytes(100);
ReadLine is a special byte[] oriented line-reader. In this case I'm
telling it "give me next 100 lines as byte[] if possible, with
line-endings stripped off." If the commons-codec project is
interested in something like this, I think ReadLine has to come along,
too.
By the way, I'm just querying the mailing list here for interest. The
Base64InputStream.java code in not-yet-commons-ssl needs javadocs
before it can come over.
And.... good work on org.apache.commons.codec.binary.Base64! I was
really astonished to see us beat openssl!
--
yours,
Julius Davies
250-592-2284 (Home)
250-893-4579 (Mobile)
http://juliusdavies.ca/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [codec] Base64InputStream
Posted by Julius Davies <ju...@gmail.com>.
Just to let people know: I'm writing a new Base64InputStream by using
the ws-commons-util logic, but rearranging things to suit this kind of
usage pattern:
#1. in = new Base64InputStream(in, DECODE);
#2. in = new Base64InputStream(in, ENCODE);
The ws-commons-util logic is really slick. I really like it. (And
it's so fast). This is a fun challenge. I'll post to the list in a
week or two with any progress.
yours,
Julius
On Sat, May 31, 2008 at 11:52 AM, Julius Davies <ju...@gmail.com> wrote:
> Hi, Jochen,
>
>
> Thanks for pointing out that method. I was looking for a static
> method. I didn't expect it to be an instance method! Anyway, now
> that I've found it, ws-commons-util is the winner!
>
>
> [16.203 seconds] ws-commons-util THE WINNER!!!!
>
> [21.946 seconds] not-yet-commons-ssl
>
>
> (For some reason not-yet-commons-ssl starting running faster....
> weird. But sun, iharder, and openssl didn't change!)
>
>
> Here's how I used ws-commons-util:
>
> FileInputStream fi = new FileInputStream(args[0]);
> InputStream in = new BufferedInputStream(fi);
> InputStreamReader reader = new InputStreamReader(in, "US-ASCII");
> char[] buf = new char[8192];
> Base64 b64 = new Base64();
> Writer w = b64.newDecoder(out);
>
> long start = System.currentTimeMillis();
> int c = reader.read(buf);
> while (c >= 0) {
> if (c > 0) {
> w.write(buf, 0, c);
> }
> c = reader.read(buf);
> }
> long duration = System.currentTimeMillis() - start;
>
>
> yours,
>
> Julius
>
>
> On Sat, May 31, 2008 at 1:18 AM, Jochen Wiedmann
> <jo...@gmail.com> wrote:
>> On Sat, May 31, 2008 at 9:06 AM, Julius Davies <ju...@gmail.com> wrote:
>>
>>> I also tried to test with ws-common, but the Base64 library in there
>>> doesn't have a streaming decoder, so I don't think it can handle a
>>> 700MB file.
>>
>> And what do you think that
>>
>> http://ws.apache.org/commons/util/apidocs/org/apache/ws/commons/util/Base64.html#newDecoder(java.io.OutputStream)
>>
>> is? For performance tuning, increasing the underlying buffer might be important.
>>
>> Jochen
>>
>>
--
yours,
Julius Davies
250-592-2284 (Home)
250-893-4579 (Mobile)
http://juliusdavies.ca/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [codec] Base64InputStream
Posted by Julius Davies <ju...@gmail.com>.
Hi, Jochen,
Thanks for pointing out that method. I was looking for a static
method. I didn't expect it to be an instance method! Anyway, now
that I've found it, ws-commons-util is the winner!
[16.203 seconds] ws-commons-util THE WINNER!!!!
[21.946 seconds] not-yet-commons-ssl
(For some reason not-yet-commons-ssl starting running faster....
weird. But sun, iharder, and openssl didn't change!)
Here's how I used ws-commons-util:
FileInputStream fi = new FileInputStream(args[0]);
InputStream in = new BufferedInputStream(fi);
InputStreamReader reader = new InputStreamReader(in, "US-ASCII");
char[] buf = new char[8192];
Base64 b64 = new Base64();
Writer w = b64.newDecoder(out);
long start = System.currentTimeMillis();
int c = reader.read(buf);
while (c >= 0) {
if (c > 0) {
w.write(buf, 0, c);
}
c = reader.read(buf);
}
long duration = System.currentTimeMillis() - start;
yours,
Julius
On Sat, May 31, 2008 at 1:18 AM, Jochen Wiedmann
<jo...@gmail.com> wrote:
> On Sat, May 31, 2008 at 9:06 AM, Julius Davies <ju...@gmail.com> wrote:
>
>> I also tried to test with ws-common, but the Base64 library in there
>> doesn't have a streaming decoder, so I don't think it can handle a
>> 700MB file.
>
> And what do you think that
>
> http://ws.apache.org/commons/util/apidocs/org/apache/ws/commons/util/Base64.html#newDecoder(java.io.OutputStream)
>
> is? For performance tuning, increasing the underlying buffer might be important.
>
> Jochen
>
>
> --
> Look, that's why there's rules, understand? So that you think before
> you break 'em.
>
> -- (Terry Pratchett, Thief of Time)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
--
yours,
Julius Davies
250-592-2284 (Home)
250-893-4579 (Mobile)
http://juliusdavies.ca/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [codec] Base64InputStream
Posted by Jochen Wiedmann <jo...@gmail.com>.
On Sat, May 31, 2008 at 9:06 AM, Julius Davies <ju...@gmail.com> wrote:
> I also tried to test with ws-common, but the Base64 library in there
> doesn't have a streaming decoder, so I don't think it can handle a
> 700MB file.
And what do you think that
http://ws.apache.org/commons/util/apidocs/org/apache/ws/commons/util/Base64.html#newDecoder(java.io.OutputStream)
is? For performance tuning, increasing the underlying buffer might be important.
Jochen
--
Look, that's why there's rules, understand? So that you think before
you break 'em.
-- (Terry Pratchett, Thief of Time)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org