You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by jo...@teamware.co.uk on 2001/10/12 12:13:00 UTC

Re:Added comments to InputStream and OutputStrea m

I asked my colleague your question on Unicode & bytes - this was his reply :

Unicode is 16 bits.  UTF-8 needs 1 byte for a 7-bit character (ASCII),
2 bytes for an 11-bit character (including ISO-8859-1), and 3 bytes for
a 16-bit character.

       DaveS


Joanne



Dmitry Serebrennikov  (11/10/2001  18:44):
>I figured that I might as well be adding comments as I am reading and
>figuring out the code.
>One thing I was not clear on - characters are stored with 1 to 3 bytes.
>Is that sufficient to represent all Unicode characters? I thought
>Unicode was four bytes.
>
>Index: InputStream.java
>===================================================================
>RCS file:
>/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/store/InputStream.
java,v
>retrieving revision 1.1.1.1


Re: Added comments to InputStream and OutputStrea m

Posted by Dmitry Serebrennikov <dm...@earthlink.net>.
>
>
>I'm a bit confused about this discussion; Java does a great job of
>hiding character encodings from you. Is Lucene turning byte arrays
>into character arrays somewhere?
>
Yes, but I think it does a good job.
See org.apache.lucene.store.{InputStream,OutputStream}.

>
>                                                     nelson@monkey.org
>.       .      .     .    .   .  . . http://www.media.mit.edu/~nelson/
>



Re: Re:Added comments to InputStream and OutputStrea m

Posted by Nelson Minar <ne...@monkey.org>.
>Unicode is 16 bits.  UTF-8 needs 1 byte for a 7-bit character (ASCII),
>2 bytes for an 11-bit character (including ISO-8859-1), and 3 bytes for
>a 16-bit character.

This is partly true. Unicode itself is coding independent. I believe
Unicode is currently defined as having up to 2^31 positions, although
the current plan is for somewhere between 2^20 and 2^21 characters.
(2^16 characters was the old Unicode standard - dropped when someone
pointed out that Chinese alone has more than 2^16 characters).

Unicode needs to be encoded somehow as a sequence of words. UTF-8
encodes Unicode as sequences of 8 bit words - either 1, 2, or 3
depending on the character. UTF-16 encodes it as a sequence of 16 bit
words: 1 or 2. UTF-32 encodes it as a sequence of 32 bit words, always
1 per character.

UTF-8 is the most common encoding. It handles ISO-Latin-1 easily (fits
in 1 word).

Unicode is cool - if you want to learn more, see
  http://www.unicode.org/
  http://www.unicode.org/unicode/faq/utf_bom.html


I'm a bit confused about this discussion; Java does a great job of
hiding character encodings from you. Is Lucene turning byte arrays
into character arrays somewhere?

                                                     nelson@monkey.org
.       .      .     .    .   .  . . http://www.media.mit.edu/~nelson/