You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Du...@Sun.COM on 2007/09/11 01:56:42 UTC
Questions Lucene
Dear Gurus
We are evaluating the possibliity of using Lucene for our
Searching needs. I have few questions before we embark on this project
of evaluating Lucene
1) What are the various languages supported by Lucene.? Looks
like its able to handle only English . We are trying to see if it works
with Japanese / Chinese and other characters
Can some one answer
2) After Lucene indexes a given data set, how does Lucene handle
incremental / dymanic change in the data. In other words, our data keeps
changing ; how
does Lucene handle this changing data. Does it re-index
every new file entering this data set ?. Or Does it do it index the data
in increments ?
3) How does Lucene handle deleted files from a particular data
set ?. What we are concerned is that, does Lucene automatically figure
out if a particular file is deleted from the data set ?.
and it immediately removes the index to that particular file ?
4) Please consider the following Scenario. When Lucene is given
the following files to Index.
a) Files under /xyz/abc ( Say x.txt, y.txt, a.txt, b.txt,
c.txt etc.. )
b) Files under /def/ghi ( Say none.txt, dude.txt, hello.txt
etc.. )
So after Lucene finished indexing these file under these two
directories. And a subsequent search for say a "key word" in hello.txt
is made
What does Lucene return; does it return i.e the fully
qualified location of this file ? /def/ghi/hello.txt
5) How does Lucene index a particular set of files. I.e *based*
on key words ?. Based on sentences ? Based on what criterion ?
6) is Lucene multi-threaded ?. For example if Lucene is indexing
a set of files in a given data set, and for example if there is a Huge
file ( 2 GB file ). Does Lucene index this file in parts (i.e parallely
i.e in multi-threaded fashion ? or does it index this file
sequentially
7) Also if a data set has multiple files, does Lucene process each
file seperately in a different thread ? or does it do it sequentially
8) Does lucene index only text files ?. We have few data bases is
it possible for us to Index the data in these data bases ?
9) Are there any performance Bench Marks for Lucene
Thanks a lot in Advance!!
-- Thanks
Durga Deep Tirunagari
Santaclara, CA
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by reply email
and destroy all copies of the original message.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Re: Questions Lucene
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Sep 10, 2007, at 7:56 PM, Durga.Tirunagari@Sun.COM wrote:
> 1) What are the various languages supported by Lucene.?
> Looks like its able to handle only English . We are trying to see
> if it works with Japanese / Chinese and other characters
> Can some one answer
Lucene internally uses UTF-8 (the Java modified version) so you won't
have any encoding issues. And everything is just text inside the
index, so no problem with Chinese, Japanese, or any other language
I've encountered - but certainly there are language-specific
considerations such as stemming, stop word removal, and whether to do
anything special to tokenize on "words" in non-whitespace-separated
languages such as Chinese or use n-gramming, or just simple character
tokenization.
> 2) After Lucene indexes a given data set, how does Lucene
> handle incremental / dymanic change in the data. In other words,
> our data keeps changing ; how
> does Lucene handle this changing data. Does it re-index
> every new file entering this data set ?. Or Does it do it index the
> data in increments ?
There is really no such thing as an "update" operation, so the
application is responsible for effecting that with a delete and re-
add on a per-document basis.
> 3) How does Lucene handle deleted files from a particular
> data set ?. What we are concerned is that, does Lucene
> automatically figure out if a particular file is deleted from the
> data set ?.
> and it immediately removes the index to that particular
> file ?
> 4) Please consider the following Scenario. When Lucene
> is given the following files to Index.
>
> a) Files under /xyz/abc ( Say x.txt, y.txt, a.txt, b.txt,
> c.txt etc.. )
> b) Files under /def/ghi ( Say none.txt, dude.txt,
> hello.txt etc.. )
> So after Lucene finished indexing these file under these
> two directories. And a subsequent search for say a "key word" in
> hello.txt is made
> What does Lucene return; does it return i.e the fully
> qualified location of this file ? /def/ghi/hello.txt
Lucene is about text, not files per se. It is your application that
will map that kind of logic on top of Lucene. Lucene itself knows
nothing of the files you want to index, delete, search - you will
build that mapping in yourself. Your application will be responsible
for keeping data and the index in sync.
> 5) How does Lucene index a particular set of files. I.e
> *based* on key words ?. Based on sentences ? Based on what criterion ?
Again, it doesn't deal with "files"... your application deals with
that, Lucene is handed text. As for how it makes words in text
searchable - read up on Lucene Analyzers. They break the text into
searchable terms.
> 6) is Lucene multi-threaded ?. For example if Lucene is
> indexing a set of files in a given data set, and for example if
> there is a Huge file ( 2 GB file ). Does Lucene index this file in
> parts (i.e parallely i.e in multi-threaded fashion ? or
> does it index this file sequentially
Lucene is isn't multi-threaded, but most operations are thread-safe
so you can parallelize your application to index multiple documents
simultaneously, for example. You may be able to parallelize the
parsing of those huge files but you'd need to bring that together
into a single Document instance to hand to Lucene's IndexWriter.
> 7) Also if a data set has multiple files, does Lucene process
> each file seperately in a different thread ? or does it do it
> sequentially
Again, this is up to your application entirely.
> 8) Does lucene index only text files ?. We have few data bases
> is it possible for us to Index the data in these data bases ?
See above :) All Lucene cares about is text. How you get text to
it matters not to Lucene.
> 9) Are there any performance Bench Marks for Lucene
There is a benchmarker framework built into the trunk codebase
suitable for making your own. There's some stuff here: http://
lucene.apache.org/java/docs/benchmarks.html and some good stuff
linked from http://wiki.apache.org/lucene-java/BasicsOfPerformance
that should get you started.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Questions Lucene
Posted by Steven Rowe <sa...@syr.edu>.
Hi Durga,
I have moved this discussion to the java-user list, since the java-dev
list is devoted to development of the Java Lucene library, and not to
questions about its capabilities. My answers are inline below.
Durga.Tirunagari@Sun.COM wrote:
> 1) What are the various languages supported by Lucene.? Looks
> like its able to handle only English . We are trying to see if it works
> with Japanese / Chinese and other characters
Lucene's native encoding is the Unicode character set, which covers most
languages. If you have a way to convert your documents' encoding into
Unicode, then Lucene can index them. The short answer is that Lucene
can handle most languages. However, precision and/or recall may suffer
for those languages for which Lucene does not provide any specific
resources. Read on for the long answer.
In Lucene parlance, "analysis" is the process whereby raw text is
converted to index terms, a.k.a. tokens. There are three areas of
analysis where Lucene does not fully support all languages:
tokenization, stopword lists, and stemming. Most of the "analyzers"
(processing pipelines) in Lucene include these three components.
1. Tokenization: Lucene's StandardTokenizer[1] handles languages that
use spaces between words; however, since the Chinese and Japanese
orthographies do not employ word-separating characters,
StandardTokenizer produces single-character tokens for these languages,
rather than single-word tokens. See the Lucene Sandbox for
CJKTokenizer[2], which instead produces overlapping bigram tokens.
Notably, however, there are no word-segmenting analyzers for Chinese or
Japanese in Lucene. Contributions are welcome!
2. Stopword lists: Lucene's StandardAnalyzer employs the English
stopword list in StopAnalyzer[3] to remove stopwords. There are several
analyzers in the Lucene Sandbox that include language-specific stopword
lists.[4]
3. Stemming: In the Lucene Sandbox, there are pre-compiled Snowball
stemmers for several Western languages[5].
> 2) After Lucene indexes a given data set, how does Lucene handle
> incremental / dymanic change in the data. In other words, our data keeps
> changing ; how does Lucene handle this changing data. Does it re-index every
> new file entering this data set ?. Or Does it do it index the data in
> increments ?
Lucene can be used to index incrementally. It has no document update
functionality, however; one must first delete and then re-add a modified
document. See my answer to your question #3, below, for more info.
> 3) How does Lucene handle deleted files from a particular data set
> ?. What we are concerned is that, does Lucene automatically figure out
> if a particular file is deleted from the data set ?. and it immediately
> removes the index to that particular file ?
Lucene can handle document deletion. However, Lucene's index readers
are only aware of the state of an index at the point at which they were
opened -- in order to see the changes introduced by document deletion
(and addition), one must close and re-open the index reader. This
process can be less than "immediate".
> 4) Please consider the following Scenario. When Lucene is
> given the following files to Index.
>
> a) Files under /xyz/abc ( Say x.txt, y.txt, a.txt, b.txt, c.txt
> etc.. )
> b) Files under /def/ghi ( Say none.txt, dude.txt, hello.txt etc.. )
> So after Lucene finished indexing these file under these two
> directories. And a subsequent search for say a "key word" in hello.txt
> is made
> What does Lucene return; does it return i.e the fully qualified
> location of this file ? /def/ghi/hello.txt
Lucene returns an ordered list of matching documents. Lucene documents
are each comprised of a user-specified set of "field"s. If you wish to
remember the name of the file from which a document was constructed, you
can store the filename in a field, and then retrieve this field's
contents for a document returned by a search.
> 5) How does Lucene index a particular set of files. I.e
> *based* on key words ?. Based on sentences ? Based on what criterion ?
You decide :). See the discussion above under your first question.
> 6) is Lucene multi-threaded ?. For example if Lucene is indexing a
> set of files in a given data set, and for example if there is a Huge
> file ( 2 GB file ). Does Lucene index this file in parts (i.e parallely
> i.e in multi-threaded fashion ? or does it index this file
> sequentially
The Lucene API is not multi-threaded, but it can be used in a
multi-threaded application. Unless otherwise noted in the API
documentation[6], Lucene methods should be thread-safe.
> 7) Also if a data set has multiple files, does Lucene process each
> file seperately in a different thread ? or does it do it sequentially
Again, Lucene can be used in a multi-threaded application, but it is not
itself multi-threaded. Populating a single index from multiple threads
is a standard, supported use of the Lucene API.
> 8) Does lucene index only text files ?. We have few data bases is
> it possible for us to Index the data in these data bases ?
Extracting analyzable text from original sources is not part of Lucene's
functionality. See the FAQ[7] for some information on extracting text
from different file types, and also on indexing databases[8].
> 9) Are there any performance Bench Marks for Lucene
Yes: <http://lucene.apache.org/java/docs/benchmarks.html>. Also, search
the java-user and java-dev lists.
Steve
[1] StandardTokenizer and StandardAnalyzer:
<http://svn.apache.org/repos/asf/lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/>
[2] CJKTokenizer and CJKAnalyzer:
<http://svn.apache.org/repos/asf/lucene/java/trunk/contrib/analyzers/src/java/org/apache/lucene/analysis/cjk/>
[3] StopAnalyzer:
<http://svn.apache.org/repos/asf/lucene/java/trunk/src/java/org/apache/lucene/analysis/StopAnalyzer.java>
[4] Sandbox Analyzers:
<http://svn.apache.org/repos/asf/lucene/java/trunk/contrib/analyzers/src/java/org/apache/lucene/analysis/>
[5] Sandbox Snowball stemmers:
<http://svn.apache.org/repos/asf/lucene/java/trunk/contrib/snowball/src/java/net/sf/snowball/ext/>
[6] Lucene trunk API docs:
<http://lucene.zones.apache.org:8080/hudson/job/Lucene-Nightly/javadoc/index.html>
[7] Lucene FAQ: <http://wiki.apache.org/lucene-java/LuceneFAQ>
[8] How to index a database from the Lucene FAQ:
<http://wiki.apache.org/lucene-java/LuceneFAQ#head-109358021acbfc89456e446740dc2bbf9049950f>
--
Steve Rowe
Center for Natural Language Processing
http://www.cnlp.org/tech/lucene.asp
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org