You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by jo...@teamware.co.uk on 2001/10/12 12:13:00 UTC
Re:Added comments to InputStream and OutputStrea
m
I asked my colleague your question on Unicode & bytes - this was his reply :
Unicode is 16 bits. UTF-8 needs 1 byte for a 7-bit character (ASCII),
2 bytes for an 11-bit character (including ISO-8859-1), and 3 bytes for
a 16-bit character.
DaveS
Joanne
Dmitry Serebrennikov (11/10/2001 18:44):
>I figured that I might as well be adding comments as I am reading and
>figuring out the code.
>One thing I was not clear on - characters are stored with 1 to 3 bytes.
>Is that sufficient to represent all Unicode characters? I thought
>Unicode was four bytes.
>
>Index: InputStream.java
>===================================================================
>RCS file:
>/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/store/InputStream.
java,v
>retrieving revision 1.1.1.1
Re: Added comments to InputStream and OutputStrea m
Posted by Dmitry Serebrennikov <dm...@earthlink.net>.
>
>
>I'm a bit confused about this discussion; Java does a great job of
>hiding character encodings from you. Is Lucene turning byte arrays
>into character arrays somewhere?
>
Yes, but I think it does a good job.
See org.apache.lucene.store.{InputStream,OutputStream}.
>
> nelson@monkey.org
>. . . . . . . . http://www.media.mit.edu/~nelson/
>
Re: Re:Added comments to InputStream and OutputStrea
m
Posted by Nelson Minar <ne...@monkey.org>.
>Unicode is 16 bits. UTF-8 needs 1 byte for a 7-bit character (ASCII),
>2 bytes for an 11-bit character (including ISO-8859-1), and 3 bytes for
>a 16-bit character.
This is partly true. Unicode itself is coding independent. I believe
Unicode is currently defined as having up to 2^31 positions, although
the current plan is for somewhere between 2^20 and 2^21 characters.
(2^16 characters was the old Unicode standard - dropped when someone
pointed out that Chinese alone has more than 2^16 characters).
Unicode needs to be encoded somehow as a sequence of words. UTF-8
encodes Unicode as sequences of 8 bit words - either 1, 2, or 3
depending on the character. UTF-16 encodes it as a sequence of 16 bit
words: 1 or 2. UTF-32 encodes it as a sequence of 32 bit words, always
1 per character.
UTF-8 is the most common encoding. It handles ISO-Latin-1 easily (fits
in 1 word).
Unicode is cool - if you want to learn more, see
http://www.unicode.org/
http://www.unicode.org/unicode/faq/utf_bom.html
I'm a bit confused about this discussion; Java does a great job of
hiding character encodings from you. Is Lucene turning byte arrays
into character arrays somewhere?
nelson@monkey.org
. . . . . . . . http://www.media.mit.edu/~nelson/