You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Ola Berg <ol...@ports.se> on 2002/11/15 10:06:20 UTC
Handling text encodings
I am about to develope punycode and race converters at work (both are mechanisms for representing Unicode as ASCII back and forth, used fx in multilingual domain names such as "bärs-och-öl.org"), which gave me this thought:
Text codecing is a pretty common task (commons net, Tomcat etc etc). Isn't there room for a common mechanism?
And BTW, if anyone has done (or know of) punycode and/or race codecs, maybe there is room for cooperation? I am investigating, and just found some example code from a Unicode.org meeting.
/Ó
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [codec] Handling text encodings
Posted by Ola Berg <ol...@ports.se>.
> It should work with streams, no doubt about it.
> I think that there should be two separate
> interfaces-- at least that's what I've usually done in such situations.
An argument against that would be that both en- and decoding are simply stream transformations. It is the context (or your mind or need at that particular time) that decides whether this is a decoding or encoding transformation.
In a neutral way, two transformations could be defined, and a third object (the codec) defines that transformation a is encoding while transformation b is decoding.
> (or at least providing interfaces in advance to point the way, so that everything
> will grow nicely together).
Sure, anything we come up with should be able to adapt to common stream handling routines.
What I smell is a generic interface, not belonging in codec, but in lang, for these kind of transformation.
And while I think at it, I think that one will end up with something very similar to the streams classes in io (because it is justified with block handling as well as singular symbol handling).
/O
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [codec] Handling text encodings (one more thing, sorry)
Posted by Ola Berg <ol...@ports.se>.
> Maybe we need two concepts:
>
> A ChunkCodec - like Soundex, Metaphone, Refined Soundex, Message
> digests....
> And a StreamCodec - like Base64, Rot13, compression algorithms, sound
> encoding...
Sun dealt with the IO in a block/stream neutral way, where InputStream provides a stream oriented view on block oriented media and vice versa. InputStream completely hides the underlying media's orientation. I don't like it. OTOH, I like its possibility to view it in a unified way.
Lots of munching algorithms benefits or needs chunks (Base64 for one), where the implementation has to fake a completely streams based view. Writing the algorithm in a block oriented way is easy, the problem is to interface with it.
The problem with a codec interface that takes String as input is that there are few algorithms that will benefit from taking strings of arbitrary length. Different chunk oriented algorithms works at the lowest level with different chunk sizes and types.
Ideally, you want it to be easy to write the codec, in a way that suits you. Then you want to interface with it in another way. Smells like some kind adapter task/ server provider interface.
Suggestions?
/O
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [codec] Handling text encodings (one more thing, sorry)
Posted by "O'brien, Tim" <to...@transolutions.net>.
A lot of good points. Soundex, Metaphone, and Refined Soundex all deal
with language, it would make more sense if these classes were moved into
a language subpackage.
With regards to streams, I think it makes sense for something like
Base64 - most definitely this should be a stream oriented codec. My
only question relates to something like Metaphone or Soundex. The
soundex algorithm is a truncated encoding that was primarily developed
to encode last names - for example "O'Brien", or "Varszegi". It seems
like wrapping "O'Brien" in a StringReader just to get the Soundex "O435"
is overkill. In other words, even if I had a 512 character String, I'm
still only producing a 4 character code ( unless I use Refined Soundex
).
The only reason, I bring that up is because I need to be able to Soundex
about 120,000 strings and populate a ternary search tree in a very
limited time ( 2-4 seconds ). If I had to insert a "new StringReader()"
into this process I'd imagine that I'd be waiting much longer to create
this index.
For Soundex, Metaphone, Refined Soundex, I'm more inspired by the
java.security.MessageDigest class.
Maybe we need two concepts:
A ChunkCodec - like Soundex, Metaphone, Refined Soundex, Message
digests....
And a StreamCodec - like Base64, Rot13, compression algorithms, sound
encoding...
--------
Tim O'Brien
Transolutions, Inc.
W 847-574-2143
M 847-863-7045
> -----Original Message-----
> From: Jeff Varszegi [mailto:jvarszegi@yahoo.com]
> Sent: Tuesday, November 19, 2002 4:02 AM
> To: Jakarta Commons Developers List
> Subject: [codec] Handling text encodings (one more thing, sorry)
>
>
> I also think that if there are going to be lots of codecs in
> the project over time, all the classes for a particular area
> should be in subpackages, like the Base64 codec currently is.
> That means that the Metaphone codec etc. should be moved
> down into a subpackage, and the codec package should just
> have the generic stuff.
>
> You're really getting this insomniac's seventy-five cents'
> worth tonight. ;O)
>
> -Jeff
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
[codec] Handling text encodings (one more thing, sorry)
Posted by Jeff Varszegi <jv...@yahoo.com>.
I also think that if there are going to be lots of codecs in the project over time, all the
classes for a particular area should be in subpackages, like the Base64 codec currently is. That
means that the Metaphone codec etc. should be moved down into a subpackage, and the codec package
should just have the generic stuff.
You're really getting this insomniac's seventy-five cents' worth tonight. ;O)
-Jeff
__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [codec] Handling text encodings
Posted by Jeff Varszegi <jv...@yahoo.com>.
I just wanted to point out that in similar situations, more than one XML API developer has chosen
to force String input to be wrapped in StringReader instances. This is in a whole programming
area devoted to nothing but processing text! I don't think it's inappropriate at all, even though
I've seen XML utility methods that take Strings more times than I can remember. My main point,
though, is that we all may as well code it right the first time for anything that's not trivial.
You can always wrap codec code that deals with streams to deal with chunks, but not the other way
around.
Jeff
__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [codec] Handling text encodings
Posted by Jeff Varszegi <jv...@yahoo.com>.
It should work with streams, no doubt about it. I think that there should be two separate
interfaces-- at least that's what I've usually done in such situations. You can make a separate
Encoder and Decoder interface, and a Codec interface that extends them both. That gives lots of
flexibility if you want to include everything in one class.
Check out com.sun.image.codec.jpeg; here they have separate encoder and decoder classes. I read
that stuff a while back and it flavored my thinking. Now check out out the classes in
com.sun.imageio . Everything is readers and writers. You may want to think about setting things
up this way, too (or at least providing interfaces in advance to point the way, so that everything
will grow nicely together).
Now, here's one more thing to think about: intermediate encodings. I had to write some stuff
using IBM machine-translation engines a while back. I remember thinking how dumb it was that one
needed to install a separate engine for every language pair. Lots of pairs, as you can guess,
hadn't been implemented yet, but there were presumably thousands of IBM coders hard at work
implementing the n! engines necessary to supply comprehensive coverage for the world's languages.
They all had different dictionaries, even. After that (actually, even before that time), a lot of
focus in the translation-research community was put in the translation research community on
translating to an intermediate form. Like microcrotch's CLR. Maybe we can wrassle out (without
spending too too much time) a decent way of arranging that.
Jeff
--- Ola Berg <ol...@ports.se> wrote:
> > > The codec package is very simple. Right now it contains 3 encoders
> > > specifically geared towards language ( Soundex, RefinedSoundex, and
> > > Metaphone ). It also contains a Base64 encoder and decoder.
> > >
> > > There is only one interface "Encoder" with one method "public
> > > String encode(String pString)". I think we need another interface
> > > "Decoder", with a similarly simple interface "public String decode(String
> > > pString)".
>
> Hmm, I see a couple of issues with this.
>
> 1) It encodes chunks, and not streams. This is a scalability issue.
>
> 2) It is geared towards text. For Bootstring, I need arbitrary symbols.
>
> 3) There is no need for another interface with identical signatures. Maybe a Codec class that
> points out two "coders" (one encoder and one decoder).
>
> For the short term, I think that a Punycode codec will do, and I will of course use Encoder as
> you have put it.
>
> /O
>
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [codec] Handling text encodings
Posted by Ola Berg <ol...@ports.se>.
> > The codec package is very simple. Right now it contains 3 encoders
> > specifically geared towards language ( Soundex, RefinedSoundex, and
> > Metaphone ). It also contains a Base64 encoder and decoder.
> >
> > There is only one interface "Encoder" with one method "public
> > String encode(String pString)". I think we need another interface
> > "Decoder", with a similarly simple interface "public String decode(String
> > pString)".
Hmm, I see a couple of issues with this.
1) It encodes chunks, and not streams. This is a scalability issue.
2) It is geared towards text. For Bootstring, I need arbitrary symbols.
3) There is no need for another interface with identical signatures. Maybe a Codec class that points out two "coders" (one encoder and one decoder).
For the short term, I think that a Punycode codec will do, and I will of course use Encoder as you have put it.
/O
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Handling text encodings
Posted by Henri Yandell <ba...@generationjava.com>.
On Mon, 18 Nov 2002, Tim O'Brien wrote:
> The codec package is very simple. Right now it contains 3 encoders
> specifically geared towards language ( Soundex, RefinedSoundex, and
> Metaphone ). It also contains a Base64 encoder and decoder.
>
> There is only one interface "Encoder" with one method "public
> String encode(String pString)". I think we need another interface
> "Decoder", with a similarly simple interface "public String decode(String
> pString)".
>
> Soundex, RefinedSoundex, and Metaphone are all one way Encoders, there is
> no way to "Decode" from these algorithms. Punycode, Bootstring, and
> Base64 are encoding and decoding algorithms, so I see the need for another
> Decoder algorithm.
How about adding in a OneWay and TwoWay interfaces for Encoder and
Decoder. Probably as extensions of Enc/Dec.
TwoWayEncoder would have a getDecoder method. Returning either a Decoder
or a Class.
OneWay would just be a marker.
If a Class, then EncoderUtils could be used to get a Decoder for an
Encoder.
Just ideas :)
Hen
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Handling text encodings
Posted by Tim O'Brien <to...@transolutions.net>.
The codec package is very simple. Right now it contains 3 encoders
specifically geared towards language ( Soundex, RefinedSoundex, and
Metaphone ). It also contains a Base64 encoder and decoder.
There is only one interface "Encoder" with one method "public
String encode(String pString)". I think we need another interface
"Decoder", with a similarly simple interface "public String decode(String
pString)".
Soundex, RefinedSoundex, and Metaphone are all one way Encoders, there is
no way to "Decode" from these algorithms. Punycode, Bootstring, and
Base64 are encoding and decoding algorithms, so I see the need for another
Decoder algorithm.
Tim O'Brien
tobrien@discursive.com
On Mon, 18 Nov 2002, Ola Berg wrote:
> > Sounds like a logical addition to the codec project. Also, the IETF
> > draft contains a sample implementation in C, it should be easy to
> > convert this over to Java.
> >
> > http://www.ietf.org/internet-drafts/draft-ietf-idn-punycode-03.txt
>
> Can you give me any starting points for how to implement a codec according to your framework. I plan to implement a generic Bootstring codec, and a specialized Punycode codec on top of that.
>
> /O
>
>
> > -----Original Message-----
> > From: Ola Berg [mailto:ola.berg@ports.se]
> > Sent: Friday, November 15, 2002 3:06 AM
> > To: Jakarta Commons Developers List
> > Subject: Handling text encodings
> >
> >
> > I am about to develope punycode and race converters at work
> > (both are mechanisms for representing Unicode as ASCII back
> > and forth, used fx in multilingual domain names such as
> > "bärs-och-öl.org"), which gave me this thought:
> >
> > Text codecing is a pretty common task (commons net, Tomcat
> > etc etc). Isn't there room for a common mechanism?
> >
> > And BTW, if anyone has done (or know of) punycode and/or race
> > codecs, maybe there is room for cooperation? I am
> > investigating, and just found some example code from a
> > Unicode.org meeting.
> >
> > /Ó
> >
> >
> >
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> > <mailto:commons-dev-> unsubscribe@jakarta.apache.org>
> > For
> > additional commands,
> > e-mail: <ma...@jakarta.apache.org>
> >
>
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
----------------------
Tim O'Brien
Evanston, IL
(847) 863-7045
tobrien@discursive.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [collections][SUBMIT] A faster version of FastArrayList
Posted by Stephen Colebourne <sc...@btopenworld.com>.
I am currently uncertain as to whether to commit this. The problem lies with
serialization.
The current FastArrayList holds a reference to an ArrayList, your
replacement holds references to an Object[] and a size (plus some other
things that should be transient). The question is whether we try to maintain
binary compatability for serialised object across releases. Any views ?
Otherwise the basic theory behind the changes seems sound.
Stephen
----- Original Message -----
From: "Jeff Varszegi" <jv...@yahoo.com>
> This version of FastArrayList looks to be from two to twenty times as fast
as the one currently in
> Collections. (Differences are sure to be found on different machines and
JVMs, but it does seem
> to be much faster.) I found over a twofold speed gain under the Java
1.4.0 'server' VM, as well
> as a twentyfold gain under the 1.4.0 'client' VM, on my machine (a laptop
running WinXP Pro, 512M,
> 1.2 GHz). I really like the FastArrayList class, I just thought it would
be nice if it could be
> faster.
>
> I achieved the speed increase mainly by reducing method-call overhead.
Instead of wrapping
> instances of ArrayList, I took the code from ArrayList itself and adapted
it to the FastArrayList
> concept, directly swapping out arrays instead of ArrayList instances. I
preserved the behavior of
> FastArrayList wherever appropriate, actually adding more synchronization
in a couple of places.
> I made a test script that extends TestList. I had problems with the
script, but I really suspect
> that it is with the test script this time (and/or the way I have JUnit set
up). At least, I find
> statements like the following very puzzling:
>
> 1)
testEmptyListSerialization(org.apache.commons.collections.TestFastArrayList2
)
> java.io.NotSerializableException: java.lang.Object
> 3)
testEmptyListCompatibility(org.apache.commons.collections.TestFastArrayList2
)
> java.io.FileNotFoundException:
data\test\FastArrayList2.emptyCollection.version1.obj (The system
> cannot find the path specified)
>
> The only serialization code in the class is 'implements Serializable'.
What is an .obj file? Now
> check out the following output statement from the script; for the life of
me, I can't understand
> it:
>
> 1) testListAddByIndex(org.apache.commons.collections.TestFastArrayList2)
> junit.framework.AssertionFailedError: List should equal confirmed
expected:<[0, , One, 2, Three,
> null, 4, One, 5.0, 6.0, Seven, Eight, Nine, 10, 11, 12, Thirteen, 14, 15,
16]> but was:<[0, , One,
> 2, Three, null, 4, One, 5.0, 6.0, Seven, Eight, Nine, 10, 11, 12,
Thirteen, 14, 15, 16]>
>
>
> The two lists look identical! This is awfully strange-seeming.
>
> I've thought of a way to speed up writes to the data structure even more
while it's in fast mode,
> but I haven't put it in yet. Please take a look at the attached code and
see if it looks like it
> will be useful. I spent much of my evening messing around with it, and
now I'm going to bed. The
> class is called FastArrayList2, the test class is called
TestFastArrayList2, and I also threw in
> the test-script results for good measure. The test script has a method
named compareSpeed() that
> gives a rudimentary comparison of fast-mode read speed between the two
versions.
>
> Thanks,
>
> Jeff
>
> P.S. I renamed setFast(boolean) and getFast() setFastModeEnabled(boolean)
and
> isFastModeEnabled(), but I preserved the old signatures too. JKV
>
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Web Hosting - Let the expert host your site
> http://webhosting.yahoo.com
----------------------------------------------------------------------------
----
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
[collections][SUBMIT] A faster version of FastArrayList
Posted by Jeff Varszegi <jv...@yahoo.com>.
This version of FastArrayList looks to be from two to twenty times as fast as the one currently in
Collections. (Differences are sure to be found on different machines and JVMs, but it does seem
to be much faster.) I found over a twofold speed gain under the Java 1.4.0 'server' VM, as well
as a twentyfold gain under the 1.4.0 'client' VM, on my machine (a laptop running WinXP Pro, 512M,
1.2 GHz). I really like the FastArrayList class, I just thought it would be nice if it could be
faster.
I achieved the speed increase mainly by reducing method-call overhead. Instead of wrapping
instances of ArrayList, I took the code from ArrayList itself and adapted it to the FastArrayList
concept, directly swapping out arrays instead of ArrayList instances. I preserved the behavior of
FastArrayList wherever appropriate, actually adding more synchronization in a couple of places.
I made a test script that extends TestList. I had problems with the script, but I really suspect
that it is with the test script this time (and/or the way I have JUnit set up). At least, I find
statements like the following very puzzling:
1) testEmptyListSerialization(org.apache.commons.collections.TestFastArrayList2)
java.io.NotSerializableException: java.lang.Object
3) testEmptyListCompatibility(org.apache.commons.collections.TestFastArrayList2)
java.io.FileNotFoundException: data\test\FastArrayList2.emptyCollection.version1.obj (The system
cannot find the path specified)
The only serialization code in the class is 'implements Serializable'. What is an .obj file? Now
check out the following output statement from the script; for the life of me, I can't understand
it:
1) testListAddByIndex(org.apache.commons.collections.TestFastArrayList2)
junit.framework.AssertionFailedError: List should equal confirmed expected:<[0, , One, 2, Three,
null, 4, One, 5.0, 6.0, Seven, Eight, Nine, 10, 11, 12, Thirteen, 14, 15, 16]> but was:<[0, , One,
2, Three, null, 4, One, 5.0, 6.0, Seven, Eight, Nine, 10, 11, 12, Thirteen, 14, 15, 16]>
The two lists look identical! This is awfully strange-seeming.
I've thought of a way to speed up writes to the data structure even more while it's in fast mode,
but I haven't put it in yet. Please take a look at the attached code and see if it looks like it
will be useful. I spent much of my evening messing around with it, and now I'm going to bed. The
class is called FastArrayList2, the test class is called TestFastArrayList2, and I also threw in
the test-script results for good measure. The test script has a method named compareSpeed() that
gives a rudimentary comparison of fast-mode read speed between the two versions.
Thanks,
Jeff
P.S. I renamed setFast(boolean) and getFast() setFastModeEnabled(boolean) and
isFastModeEnabled(), but I preserved the old signatures too. JKV
__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com
Re: Handling text encodings
Posted by Ola Berg <ol...@ports.se>.
> Sounds like a logical addition to the codec project. Also, the IETF
> draft contains a sample implementation in C, it should be easy to
> convert this over to Java.
>
> http://www.ietf.org/internet-drafts/draft-ietf-idn-punycode-03.txt
Can you give me any starting points for how to implement a codec according to your framework. I plan to implement a generic Bootstring codec, and a specialized Punycode codec on top of that.
/O
> -----Original Message-----
> From: Ola Berg [mailto:ola.berg@ports.se]
> Sent: Friday, November 15, 2002 3:06 AM
> To: Jakarta Commons Developers List
> Subject: Handling text encodings
>
>
> I am about to develope punycode and race converters at work
> (both are mechanisms for representing Unicode as ASCII back
> and forth, used fx in multilingual domain names such as
> "bärs-och-öl.org"), which gave me this thought:
>
> Text codecing is a pretty common task (commons net, Tomcat
> etc etc). Isn't there room for a common mechanism?
>
> And BTW, if anyone has done (or know of) punycode and/or race
> codecs, maybe there is room for cooperation? I am
> investigating, and just found some example code from a
> Unicode.org meeting.
>
> /Ó
>
>
>
>
>
>
> --
> To unsubscribe, e-mail:
> <mailto:commons-dev-> unsubscribe@jakarta.apache.org>
> For
> additional commands,
> e-mail: <ma...@jakarta.apache.org>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Handling text encodings
Posted by "O'brien, Tim" <to...@transolutions.net>.
Sounds like a logical addition to the codec project. Also, the IETF
draft contains a sample implementation in C, it should be easy to
convert this over to Java.
http://www.ietf.org/internet-drafts/draft-ietf-idn-punycode-03.txt
--------
Tim O'Brien
Transolutions, Inc.
W 847-574-2143
M 847-863-7045
> -----Original Message-----
> From: Ola Berg [mailto:ola.berg@ports.se]
> Sent: Friday, November 15, 2002 3:06 AM
> To: Jakarta Commons Developers List
> Subject: Handling text encodings
>
>
> I am about to develope punycode and race converters at work
> (both are mechanisms for representing Unicode as ASCII back
> and forth, used fx in multilingual domain names such as
> "bärs-och-öl.org"), which gave me this thought:
>
> Text codecing is a pretty common task (commons net, Tomcat
> etc etc). Isn't there room for a common mechanism?
>
> And BTW, if anyone has done (or know of) punycode and/or race
> codecs, maybe there is room for cooperation? I am
> investigating, and just found some example code from a
> Unicode.org meeting.
>
> /Ó
>
>
>
>
>
>
> --
> To unsubscribe, e-mail:
> <mailto:commons-dev-> unsubscribe@jakarta.apache.org>
> For
> additional commands,
> e-mail: <ma...@jakarta.apache.org>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>