You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Marvin Humphrey <ma...@rectangular.com> on 2006/05/17 06:35:35 UTC
Hacking Luke for bytecount-based strings
Greets,
There does not seem to be a lot of demand for one implementation of
Lucene to read indexes generated by another implementation of Lucene
for the purposes of indexing or searching. However, there is a
demand for index browsing via Luke.
It occurred to me today that if Luke were powered by a version of
Lucene with my bytecount-based-strings patch applied, it would be
able to read indexes generated by Ferret. Ironically, it wouldn't be
able to read KinoSearch indexes unless I reverted the change which
causes the term vectors to be stored in the .fdt file. I'd probably
do that. Luke is great.
One possibility for distributing such a beast is to offer a patched
jar for download from my website. Before I start down that road,
though, I thought I'd bring up the subject here.
Thoughts?
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Hacking Luke for bytecount-based strings
Posted by Marvin Humphrey <ma...@rectangular.com>.
On May 17, 2006, at 2:04 PM, Doug Cutting wrote:
>> Detecting invalidly encoded text later doesn't help anything in
>> and of itself; lifting the requirement that everything be
>> converted to Unicode early on opens up some options.
>
> How useful are those options? Are they worth the price?
> Converting to unicode early permits one to, e.g., write encoding-
> independent tokenizers, stemmers, etc. That seems like a lot to
> throw away.
Fair enough. For Java Lucene, the main benefits of encoding
flexibility would accrue when A) your material takes up a lot more
space in UTF-8 than in another alternative, or B) you prefer a native
encoding to Unicode, most often because of the Han unification
controversy.
The space issue could be addressed by allowing UTF-16 as an
alternative. Catering to arbitrary encodings doesn't offer that much
benefit for the price, though your perspective on that may differ if
you're, say, Japanese.
>>> UTF-8 has the property that bytewise lexicographic order is the
>>> same as Unicode character order.
>> Yes. I'm suggesting that an unpatched TermBuffer would have
>> problems with my index with corrupt character data because the
>> sort order by bytestring may not be the same as sort order by
>> Unicode code point.
>
> I think you're saying that bytewise comparisons involving invalid
> UTF-8 may differ from comparisons of the unicode code points they
> represent. But if they're invalid, they don't actually represent
> unicode code points, so how can they be compared?
Repairing an invalid Unicode sequence, whether UTF-8, UTF-16BE, or
other, generally means swapping in U+FFFD "REPLACEMENT CHARACTER",
provided that you don't throw a fatal error. U+FFFD has a numeric
value which affects sort order, and the swap may also affect the
length of the sequence.
More generally, if you map from valid data in another encoding to
Unicode, lexical sorting of the source bytestring and lexical sorting
of the Unicode target will often produce differing results.
It really messes up a TermInfosReader to have terms out of sequence.
And unfortunately I misremembered how the cached Terms from the
auxiliary term dictionary get compared -- those use term.compareTo
(otherTerm) rather than termBuffer.compareTo(otherTermBuffer). The
patched version of Lucene doesn't change that, so if an invalidly
encoded term with a replacement character happens to fall on an index
point, bad things will happen.
That means the current patch is inadequate for dealing with
KinoSearch 0.05 or Ferret indexes unless the application developer
forced UTF-8 at index-time. I'd need to make additional changes in
order to guarantee that a patched Luke would work -- TermInfosReader
would need to cache the bytestrings and compare those instead.
That's effectively what KinoSearch does.
That's probably a good idea anyway, as it cuts down the RAM
requirements for caching the Term Infos Index -- so long as your data
occupies less space as a bytestring than as Java chars.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Hacking Luke for bytecount-based strings
Posted by Doug Cutting <cu...@apache.org>.
Marvin Humphrey wrote:
> I *think* that whether it was invalidly encoded or not wouldn't impact
> searching -- it doesn't in KinoSearch. It should only affect display.
I think Java's approach of converting everything to unicode internally
is useful. One must still handle dirty input, but it easy to write
output that conforms to standards. I'd hate to lose that.
Java programs have a good reputation for supporting
internationalization, better than those written in languages that
primarily represent strings as byte arrays and library utilities for
handling encodings and character sets. Java's choice of 16-bit
characters may have been an error, but the general approach of
converting all textual data to unicode internally has led to fewer
internationalization issues than are common in other systems.
> Detecting invalidly encoded text later doesn't help anything in and of
> itself; lifting the requirement that everything be converted to Unicode
> early on opens up some options.
How useful are those options? Are they worth the price? Converting to
unicode early permits one to, e.g., write encoding-independent
tokenizers, stemmers, etc. That seems like a lot to throw away.
>> UTF-8 has the property that bytewise lexicographic order is the same
>> as Unicode character order.
>
>
> Yes. I'm suggesting that an unpatched TermBuffer would have problems
> with my index with corrupt character data because the sort order by
> bytestring may not be the same as sort order by Unicode code point.
I think you're saying that bytewise comparisons involving invalid UTF-8
may differ from comparisons of the unicode code points they represent.
But if they're invalid, they don't actually represent unicode code
points, so how can they be compared?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Hacking Luke for bytecount-based strings
Posted by Marvin Humphrey <ma...@rectangular.com>.
On May 17, 2006, at 11:08 AM, Doug Cutting wrote:
> Marvin Humphrey wrote:
>> What I'd like to do is augment my existing patch by making it
>> possible to specify a particular encoding, both for Lucene and Luke.
>
> What ensures that all documents in fact use the same encoding?
In KinoSearch at this moment, zilch. Lucene would still need to read
stuff into Java chars then write it out using the specified
encoding. If we opt for output buffering rather than output counting
(the patch currently does counting, but that would have to change if
we're flexible about encoding in the index), then string.getBytes
(encoding) would guarantee it.
> The current approach of converting everything to Unicode and then
> writing UTF-8 to indexes makes indexes portable and simplifies the
> construction of search user interfaces, since only indexing code
> needs to know about other character sets and encodings.
Sure. OTOH, it's not so good for CJK users. I also opted against it
in KinoSearch because A) compatibility with the current Java Lucene
file format wasn't going to happen anyway, and B) not all Perlers use
or require valid UTF-8. I've considered adding a UTF8Enforcer
Analyzer subclass, but it hasn't been an issue. Right now, if your
source docs are mucked up, they'll be mucked up when you retrieve
them after searching. If you want to fix that, you preprocess.
Ensuring consistent encoding is the application developer's
responsibility.
> If a collection has invalidly encoded text, how does it help to
> detect that later rather than sooner?
I *think* that whether it was invalidly encoded or not wouldn't
impact searching -- it doesn't in KinoSearch. It should only affect
display. Detecting invalidly encoded text later doesn't help
anything in and of itself; lifting the requirement that everything be
converted to Unicode early on opens up some options.
>> Searches will continue to work regardless because the patched
>> Termbuffer compares raw bytes. (A comparison based on
>> Term.compareTo () would likely fail because raw bytes translated
>> to UTF-8 may not produce the same results.)
>
> UTF-8 has the property that bytewise lexicographic order is the
> same as Unicode character order.
Yes. I'm suggesting that an unpatched TermBuffer would have problems
with my index with corrupt character data because the sort order by
bytestring may not be the same as sort order by Unicode code point.
However, the patched TermBuffer uses compareBytes() rather than
compareChars(), so TermInfosReader should work fine.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
public final int compareTo(TermBuffer other) {
if (field == other.field) // fields are interned
- return compareChars(text, textLength, other.text,
other.textLength);
+ return compareBytes(bytes, bytesLength, other.bytes,
other.bytesLength);
else
return field.compareTo(other.field);
}
- private static final int compareChars(char[] v1, int len1,
- char[] v2, int len2) {
+ private static final int compareBytes(byte[] bytes1, int len1,
+ byte[] bytes2, int len2) {
int end = Math.min(len1, len2);
for (int k = 0; k < end; k++) {
- char c1 = v1[k];
- char c2 = v2[k];
- if (c1 != c2) {
- return c1 - c2;
+ int b1 = (bytes1[k] & 0xFF);
+ int b2 = (bytes2[k] & 0xFF);
+ if (b1 != b2) {
+ return b1 - b2;
}
}
return len1 - len2;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Hacking Luke for bytecount-based strings
Posted by Doug Cutting <cu...@apache.org>.
Marvin Humphrey wrote:
> What I'd like to do is augment my existing patch by making it possible
> to specify a particular encoding, both for Lucene and Luke.
What ensures that all documents in fact use the same encoding?
The current approach of converting everything to Unicode and then
writing UTF-8 to indexes makes indexes portable and simplifies the
construction of search user interfaces, since only indexing code needs
to know about other character sets and encodings.
If a collection has invalidly encoded text, how does it help to detect
that later rather than sooner?
> Searches
> will continue to work regardless because the patched Termbuffer
> compares raw bytes. (A comparison based on Term.compareTo () would
> likely fail because raw bytes translated to UTF-8 may not produce the
> same results.)
UTF-8 has the property that bytewise lexicographic order is the same as
Unicode character order.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Hacking Luke for bytecount-based strings
Posted by Marvin Humphrey <ma...@rectangular.com>.
On May 16, 2006, at 11:58 PM, Paul Elschot wrote:
> Try and invoke luke with the a lucene jar of your choice on the
> classpath before luke itself:
>
> java -cp lucene-core-1.9-rc1-dev.jar:lukeall.jar org.getopt.luke.Luke
I tried this on an index built with KinoSearch 0.05, which pre-dates
the addition of term vectors to .fdt. After working out a
SecurityException by using individual components rather than
lukeall.jar...
Luke powered by the patched library worked; Luke powered by straight-
up Lucene did not.
The source material was stuff from Wikipedia, which contains a bunch
of invalid UTF-8. KinoSearch doesn't care about that, so it's in
there in the index. No problems. :)
What I'd like to do is augment my existing patch by making it
possible to specify a particular encoding, both for Lucene and Luke.
Searches will continue to work regardless because the patched
Termbuffer compares raw bytes. (A comparison based on Term.compareTo
() would likely fail because raw bytes translated to UTF-8 may not
produce the same results.) That way, say, a Russian user who had
built a KinoSearch index using KOI8-R (assumming I revert the .fdt
change) could specify KOI8-R and have Luke display the correct
characters. Ideally, you'd want to store the index's encoding in the
index somewhere, but Lucene doesn't have a place for that, so I need
to patch both Luke and Lucene.
I wonder how Lucene would perform with my patch applied if the
indexer were spec'd to use Latin1 rather than UTF-8... patches to
the segment merging apparatus would be required...
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Hacking Luke for bytecount-based strings
Posted by Paul Elschot <pa...@xs4all.nl>.
On Wednesday 17 May 2006 06:35, Marvin Humphrey wrote:
> Greets,
>
> There does not seem to be a lot of demand for one implementation of
> Lucene to read indexes generated by another implementation of Lucene
> for the purposes of indexing or searching. However, there is a
> demand for index browsing via Luke.
>
> It occurred to me today that if Luke were powered by a version of
> Lucene with my bytecount-based-strings patch applied, it would be
> able to read indexes generated by Ferret. Ironically, it wouldn't be
> able to read KinoSearch indexes unless I reverted the change which
> causes the term vectors to be stored in the .fdt file. I'd probably
> do that. Luke is great.
>
> One possibility for distributing such a beast is to offer a patched
> jar for download from my website. Before I start down that road,
> though, I thought I'd bring up the subject here.
>
> Thoughts?
Try and invoke luke with the a lucene jar of your choice on the
classpath before luke itself:
java -cp lucene-core-1.9-rc1-dev.jar:lukeall.jar org.getopt.luke.Luke
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: Hacking Luke for bytecount-based strings
Posted by Robert Engels <re...@ix.netcom.com>.
While you're at it, why not rewrite Luke in Perl as well...
Seems like a great use of your time.
-----Original Message-----
From: Marvin Humphrey [mailto:marvin@rectangular.com]
Sent: Tuesday, May 16, 2006 11:36 PM
To: java-dev@lucene.apache.org
Cc: Andrzej Bialecki
Subject: Hacking Luke for bytecount-based strings
Greets,
There does not seem to be a lot of demand for one implementation of
Lucene to read indexes generated by another implementation of Lucene
for the purposes of indexing or searching. However, there is a
demand for index browsing via Luke.
It occurred to me today that if Luke were powered by a version of
Lucene with my bytecount-based-strings patch applied, it would be
able to read indexes generated by Ferret. Ironically, it wouldn't be
able to read KinoSearch indexes unless I reverted the change which
causes the term vectors to be stored in the .fdt file. I'd probably
do that. Luke is great.
One possibility for distributing such a beast is to offer a patched
jar for download from my website. Before I start down that road,
though, I thought I'd bring up the subject here.
Thoughts?
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org