You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Itamar Syn-Hershko <it...@divrei-tora.com> on 2008/01/21 20:59:27 UTC
Lucene, HTML and Hebrew
Hi all,
I'm starting in the process of creating Hebrew support for Lucene.
Specifically I'm using Clucene (which is an awesome and strong port), but
that shouldn't matter for my questions. Please, if you know of any info or
similar project let me know, it can save me loads of time and headaches. As
far as I could tell, such a thing is not in existance. Since I'm using
Clucene, the following questions reffer to Lucene < 2.0, should it make any
difference.
In my project I'm indexing HTML files which are either in English or Hebrew,
but may have words of the other language, and I need to enable searches for
both.
So in my application I load the HTML files one by one, taking text snippets
paragraph by paragraph (<p> tag). For each paragraph I strip it from HTML
tags and resolve HTML entities and then create a tokenized and indexed, not
stored, field. The HTML files are articles with main header, a few
sub-headers, quotes, normal text, and footnotes. I want to index all text as
normal, except for terms from headers which I want to boost (h1 the highest
boost, h2 less and so on). Footnotes should get negative boost (less than
normal text, which is neutral), quotes being indexed as field "quote" to
allow for quotes searches but are considered as normal on queries.
Due to the structure of the HTML files I have, calls to add new fields to a
document while parsing the HTML will be made in this order (psuedo code):
Doc->add(new Field("h1", "Article Name / Adam Smith",
Field::STORE_NO | Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some introduction text",
Field::STORE_NO | Field::INDEX_TOKENIZED);
Doc->add(new Field("h2", "sub-header 1", Field::STORE_NO |
Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph1", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph2", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph3", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph4", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Doc->add(new Field("h2", "sub-header 2", Field::STORE_NO |
Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph1", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph2", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph3", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph4", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Doc->add(new Field("h2", "sub-header 3", Field::STORE_NO |
Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph1", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph2", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph3", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Doc->add(new Field("normal", "some text paragrph4", Field::STORE_NO
| Field::INDEX_TOKENIZED);
Based on the above info:
1) How would Lucene treat the "normal" paragraph when they are added that
way? Would proximity and frequency data be computed between paragraph1 and
paragraph2 (last word of former with first word of latter)? What about
proximity data between "h2" paragraph and the "normal" before or after it?
2) How would I set the boosts for the headers and footnotes? I'd rather have
it stored within the index file than have to append it to each and every
query I will execute, but I'm open to suggestions. I'm more interested in
performance and flexibility.
3) When executing a query against the above-mentioned index, how would I
execute a set of words as a query (boolean quey using list of inflated
words) without repeating this set of words for each and every field? Any
support for something like *:word1 OR word2 OR word3 (instead of
normal:(word1 OR word2 OR word3) AND quote:(word1 OR word2 OR word3) AND
h1:(word1 OR word2 OR word3) etc...)?
4) Writing a Hebrew analyzer, I'm considering using a StandardAnalyzer and
just ammend it so when it recognizes a Hebrew word it will call some
function that will parse it correctly (there are some differences, such as
quotes in the middle of a word are legitmate, also remove Niqqud). If it's a
non-Hebrew word it will just continue as usual with its default behavior and
functions.
Also, this means I will index ALL words (Hebrew AND English) into the same
index. The thinking behind this is to allow for searches with both Hebrew
and English words to be performed successfully, taking into account there
shouldn't be any downsides for indexing two languages within one index. I'm
aware of the way Lucene stores words (not the whole word, but only the part
that is different from the previous), but I really don't see how bad that's
gonna be...
5) Where should a stemmer be used? As far as I see it, it should only be
used for query inflation, am I right?
I know that might be quite heavy on you guys, but I will appreciate detailed
answers. Thanks in advance!
Itamar.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Lucene, HTML and Hebrew
Posted by Itamar Syn-Hershko <it...@divrei-tora.com>.
OK, I've been processing things for a while. I came up with an idea that I
want your advice on -- is there a way I could stem the Hebrew words in my
analyzer yet keep a note of some sort of the original term which was
assembled by this stem, WITHOUT affecting frequency/proximity data? This is
I guess you can say a "must" for Hebrew words searches if one wants to make
them quick efficient and precise, but can be very useful for stem-based
indexing in other languages.
Here are some more details:
Since stemming in Hebrew is ugly - loads of abiguities, and a final stem
resulting in a 3 letters word (which can be constructed in many other forms
later -- and that's in the good case where the stemmer is decided on the
correct stem), I've been looking for some alternative. So it was either
stemming differently (finding a way to stem to larger, more descriptive
stems), or not stemming at all and using query inflation (which should yield
much better, more relevant hits, but have perfomance impact, plus can grow
to more than a few hundreds terms per word).
I'm still brainstorming on the best way to achieve top-notch Hebrew indexing
and searching with Lucene. While doing so, I've had an idea -- assuming the
following is a the IndexReader workflow (more or less):
1. Access Index terms list, find the reference to the records for each word
in Terms
2. Access the Term record in the Index, load all occurences of it (Doc ID,
Field ID along with Freq and Prox data) and do the math and scoring
according to the Query logic.
Is it possible so that for every occurrence of a word in that array
mentioned above (you probably have a name for it...) I would store the
original word used to form that stem of which this hit referrs to? So for
example if I look for "got" it becomes "get" when passed through a stemmer
and then can be found by using any of the words built from the stem "get" --
"gotten", "got", "gets" and whatever. Now, looking at that array of
occurrences I know the stem "get" was found in a specific document, but if I
wanted to score the original word higher than its sibling words built from
the same stem, having the original word stored along with the occurrence
data will certainly help (not only for scoring though - also for specific
searches where I will require the exact word to be found only).
As I said, for Hebrew this will greatly help.
Is it possible using Lucene as it is? If not, I guess this will require file
format change? And if so, can this be done using some sort of deriviation of
existing classes, or it needs to be hacked in the existing code?
If you have any idea on how to achieve similar results in more convinient
ways, please with all means do let me know...
Itamar.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Lucene, HTML and Hebrew
Posted by Steven A Rowe <sa...@syr.edu>.
On 01/22/2008 at 8:49 PM, Grant Ingersoll wrote:
> On Jan 22, 2008, at 6:06 PM, Steven A Rowe wrote:
> > On 01/21/2008 at 2:59 PM, Itamar Syn-Hershko wrote:
> > > 2) How would I set the boosts for the headers and footnotes?
> > > I'd rather have it stored within the index file than have to
> > > append it to each and every query I will execute, but I'm
> > > open to suggestions. I'm more interested in performance and
> > > flexibility.
> >
> > AFAIK, there is no way currently in Lucene to set index-time per-
> > field boosts - only per-document boosts are supported.
>
> I'm not following this part. Is this different from Field.setBoost()?
Thanks, Grant, you're absolutely right.
I stand corrected: Lucene does, in fact, store per-field boosts at index-time[1], and then uses them at query-time to score documents. As Grant indicates, the Field.setBoost() method is how to set them.
(It would be nice if Lucene had a facility to set a global field boost for all the documents in an index.)
Itamar, the details of how field boosts are used are worth checking out (the link below is to the 1.4.3 version):
<http://lucene.apache.org/java/1_4_3/api/org/apache/lucene/document/Field.html#setBoost(float)>
In particular, this part is less than obvious:
If a document has multiple fields with the same
name, all such values are multiplied together.
So if you were to add multiple Field instances with the same name to a document, as you do in your example, the field boost for the document is the *product* of the boosts set on each Field instance added to the document. One way to get around this problem is to call setBoost() just once on each document for each of the different field names. This works because the default boost assigned to the other instances is 1.
Steve
[1] It's worth noting, as the above-linked documentation for Field.setBoost() does, that field boosts are not stored independently of other normalization factors in the index.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene, HTML and Hebrew
Posted by Grant Ingersoll <gs...@apache.org>.
On Jan 22, 2008, at 6:06 PM, Steven A Rowe wrote:
>
>> 2) How would I set the boosts for the headers and footnotes?
>> I'd rather have it stored within the index file than have to
>> append it to each and every query I will execute, but I'm
>> open to suggestions. I'm more interested in performance and
>> flexibility.
>
> AFAIK, there is no way currently in Lucene to set index-time per-
> field boosts - only per-document boosts are supported.
>
I'm not following this part. Is this different from Field.setBoost()?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Lucene, HTML and Hebrew
Posted by Steven A Rowe <sa...@syr.edu>.
Hi Itamar,
On 01/24/2008 at 2:55 PM, Itamar Syn-Hershko wrote:
> > Lucene does not store proximity relations between data in different
> > fields, only within individual fields
>
> So are 2 calls for doc->add with the same field but different
> texts are considered as 1 field (latter call being internally
> appended into the former, merged into one field), or as two
> instances of the same field which do not share proximity and
> frequency data? As it seems from what you wrote later in your
> response, it seems the case is the former.
Yes. From <http://lucene.apache.org/java/1_4_3/api/org/apache/lucene/document/Document.html#add(org.apache.lucene.document.Field)>:
Adds a field to a document. Several fields may be
added with the same name. In this case, if the
fields are indexed, their text is treated as though
appended for the purposes of search.
> How can I inhibit this appending -- are there any more approaches than
> appending an invalid string like "$$$"?
Here's an idea, though it is entirely untested and may be completely false :) :
Lucene's Tokenizers are fed a Reader (in Java - I don't know about CLucene's setup, but I assume the interface is similar) and emit Tokens. Assuming that each field value from same-named fields gets its own Reader, then you could create a custom Tokenizer that, for the first Token it emits, sets a position increment greater than one - in so doing, phrase matching across same-named field values should be inhibited:
<http://lucene.apache.org/java/1_4_3/api/org/apache/lucene/analysis/Token.html#setPositionIncrement(int)>
> I've been thinking about this a bit, and I think I'd go with
> one big field for all the content, and I'd want to incorporate
> the headers into it as well. How would I boost those specific
> words - so the content field can contain both all words and
> all headers in their original order (for proximity and
> frequency data to be valid), yet keep the terms which were
> originally in a header or a sub-header boosted?
Like I wrote in a previous response:
> > One very coarse-grained boosting trick you could use is to
> > repeat the text of headers, etc., that you want to boost.
This trick adjusts the term frequency of important terms.
I don't know of any other approaches besides this trick, except using field boosting, which would require you to have separate fields.
> So, to overcome the challenges above, I was thinking about the query
> inflation approach, having a negative boost for the inflated
> terms as you suggested.
Actually, I was referring to a reduced, but non-negative, boost - like 0.5 instead of 1.0. AFAIK, Lucene does not support negative boosts.
> I will appreciate any different takes on this one, as this is
> going to be the first public Lucene Hebrew analyzer...
One thought - for ambiguous terms, your stemming component could emit all of the alternatives at the same position.
> Using this approach I only need to make sure I do not inflate those too
> much (1024 is the standard limit, right?).
1024 is the default maximum number of BooleanClause children, but you can set this higher (or lower) should you desire:
<http://lucene.apache.org/java/1_4_3/api/org/apache/lucene/search/BooleanQuery.html#setMaxClauseCount(int)>
> Also, how can I check whether a word I inflated exists in the
> index BEFORE executing the query? Is that recommended at all?
See IndexReader.terms():
<http://lucene.apache.org/java/1_4_3/api/org/apache/lucene/index/IndexReader.html#terms()>
If, as an offline process, you were to trim your query expansion map so that it included only terms known to be in the index, the resulting simpler queries should impact positively on performance.
> > It's worth noting, as the above-linked documentation for
> > Field.setBoost() does, that field boosts are not stored
> > independently of other normalization factors in the index.
>
> Does this mean I should stick with boosting fields in the
> query phase only?
No - I mentioned this only to alert you to the fact that field boosts are stored in the index only as part of the field "norm", which is an amalgam including a couple of other factors.
Index-time field boosting could potentially do good things for you - it's worth trying out.
Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Lucene, HTML and Hebrew
Posted by Itamar Syn-Hershko <it...@divrei-tora.com>.
Steve and all,
I didn't know whether to send a detailed description of my case to aid with
seeing the whole picture, or to send a list of short questions which will
require loads of follow-up. I guess I know what is better now, thanks....
>> Lucene does not store proximity relations between data in different
fields, only within individual fields
So are 2 calls for doc->add with the same field but different texts are
considered as 1 field (latter call being internally appended into the
former, merged into one field), or as two instances of the same field which
do not share proximity and frequency data?
As it seems from what you wrote later in your response, it seems the case is
the former. How can I inhibit this appending -- are there any more
approaches than appending an invalid string like "$$$"?
I've been thinking about this a bit, and I think I'd go with one big field
for all the content, and I'd want to incorporate the headers into it as
well. How would I boost those specific words - so the content field can
contain both all words and all headers in their original order (for
proximity and frequency data to be valid), yet keep the terms which were
originally in a header or a sub-header boosted? This can be a good practice
for boosting bolded or italic text in the normal paragrphs as well (only
with a lower boost).
>> Generally, stemming increases recall (proportion of matching relevant
docs among relevant docs in the entire corpus), and decreases precision
(proportion of relevant docs among matching docs).
Thats a great definition, thanks.
I'm trying to think this through, since Hebrew is not a regular case. If you
will google for Hebrew and Stemming you will get pages which talk about how
complicated is Hebrew compared to English and other European languages.
[ Warning: technical data, questions follow after this paragraph -- to
comply with the 30-seconds rule :) ]
This is extremely difficult since unique features Hebrew has like
Niqqud-less spelling (which causes many words to have several spelling
options, only one legal but the others too-common to ignore) and
three-letter stems which have many deriviations. Furthermore, English words
like and, that, of, to etc. in Hebrew are represented as one letter appended
to the beginning of the word, forming a whole new word than the original.
Discarding them while indexing is not a smart move since one would try and
look for specific term *with* this initial and would not expect results
without. Furthermore, some words which uses these initials have another
meaning when pronounced differently (like KLBI - could be read as Ke-libi
["as my heart"] where I can omit the leading K, and also as Kalbi ["My dog"]
where I cannot.
So, to overcome the challenges above, I was thinking about the query
inflation approach, having a negative boost for the inflated terms as you
suggested. I will appreciate any different takes on this one, as this is
going to be the first public Lucene Hebrew analyzer... Using this approach I
only need to make sure I do not inflate those too much (1024 is the standard
limit, right?).
Also, how can I check whether a word I inflated exists in the index BEFORE
executing the query? Is that recommended at all? -- I'm looking for the most
efficient way so search speed will still be measured in few m/s, as it is
now. The idea is to prevent, or minimize, the use of a dictionary, and
keeping the stemmer as simple as possible (and by that produce invalid words
and eliminate them before executing the search).
>> It's worth noting, as the above-linked documentation for Field.setBoost()
does, that field boosts are not stored independently of other normalization
factors in the index.
Does this mean I should stick with boosting fields in the query phase only?
Itamar.
-----Original Message-----
From: Steven A Rowe [mailto:sarowe@syr.edu]
Sent: Wednesday, January 23, 2008 1:06 AM
To: java-user@lucene.apache.org
Subject: RE: Lucene, HTML and Hebrew
Hi Itamar,
In another thread, you wrote:
> Yesterday I sent an email to this group querying about some very
> important (to me...) features of Lucene. I'm giving it another chance
> before it goes unnoticed or forgotten. If it was too long please let
> me know and I will email a shorter list of questions....
I think I have something like a 30-second rule for posts on this list: if I
can't figure out what the question is within 30 seconds, I move on. Your
post was so verbose that I gave up before I asked myself whether I could
help. (Déjà vu - upon re-reading this paragraph, it sounds very much like
something Hoss has said on this list...)
Although I answer your original post below, please don't take this as
affirmation of your "reminder" approach. In my experience, this strategy is
interpreted as badgering, and tends to affect response rate in the opposite
direction to that intended.
Short, focused questions will maximize the response rate here (and
elsewhere, I suspect). Also, it helps if there is some indication that the
questioner has attempted to answer the question for themselves using readily
available resources, but failed.
On 01/21/2008 at 2:59 PM, Itamar Syn-Hershko wrote:
> 1) How would Lucene treat the "normal" paragraph when they are added
> that way? Would proximity and frequency data be computed between
> paragraph1 and paragraph2 (last word of former with first word of
> latter)? What about proximity data between "h2" paragraph and the
> "normal" before or after it?
Lucene does not store proximity relations between data in different fields,
only within individual fields. Similarly, term frequencies are stored
per-field, not per-document.
> 2) How would I set the boosts for the headers and footnotes?
> I'd rather have it stored within the index file than have to append it
> to each and every query I will execute, but I'm open to suggestions.
> I'm more interested in performance and flexibility.
AFAIK, there is no way currently in Lucene to set index-time per-field
boosts - only per-document boosts are supported.
One very coarse-grained boosting trick you could use is to repeat the text
of headers, etc., that you want to boost, e.g.:
Doc->add(new Field("h2", "sub-header 1 $$$ sub-header 1",
Field::STORE_NO | Field::INDEX_TOKENIZED);
I included "$$$" as an example of how to break proximity between the first
and last terms in the "sub-header 1" text - note, however, that this
particular string may not serve this function properly, depending on the
analyzer you choose.
Note also that this is an issue elsewhere for you, since each addition of
field information is understood by Lucene as contiguous. That is, unless
you do something to inhibit it, proximity matches will occur between the
last term from one <h2> tag and the first term from the next <h2> tag in the
same doc.
> 3) When executing a query against the above-mentioned index, how would
> I execute a set of words as a query (boolean quey using list of
> inflated words) without repeating this set of words for each and every
> field? Any support for something like *:word1 OR word2 OR word3
> (instead of normal:(word1 OR
> word2 OR word3) AND quote:(word1 OR word2 OR word3) AND
> h1:(word1 OR word2 OR word3) etc...)?
MultiFieldQueryParser may do something like what you want:
<http://lucene.apache.org/java/1_4_3/api/org/apache/lucene/queryParser/Multi
FieldQueryParser.html>
> 4) Writing a Hebrew analyzer, I'm considering using a StandardAnalyzer
> and just ammend it so when it recognizes a Hebrew word it will call
> some function that will parse it correctly (there are some
> differences, such as quotes in the middle of a word are legitmate,
> also remove Niqqud).
> If it's a non-Hebrew word it will just continue as usual with its
> default behavior and functions.
> Also, this means I will index ALL words (Hebrew AND English) into the
> same index. The thinking behind this is to allow for searches with
> both Hebrew and English words to be performed successfully, taking
> into account there shouldn't be any downsides for indexing two
> languages within one index. I'm aware of the way Lucene stores words
> (not the whole word, but only the part that is different from the
> previous), but I really don't see how bad that's gonna be...
Not sure what the question is here - if you mean to ask "What are the
impacts of including terms from two languages in a single index?" then my
answer is "it depends"...
For languages that share orthographies (e.g. Spanish and French, to a great
extent), cognates (i.e. the same term meaning completely different things in
the two languages) could cause degraded precision. AFAIK, this is not an
issue for Hebrew and English.
The only other issue I can think of is that you will be taking symbols
(words) from two completely different meaning-systems and merging them into
the same index space. For similar contexts ("should I have separate fields
for each unit of information?"), the advice generally given on this list is
to put everything into a single field. In short: try the simplest thing
first, test, and if the performance is not good enough, then increase the
complexity of your solution, test, and iterate until it is. But you
probably already knew that :).
> 5) Where should a stemmer be used? As far as I see it, it should only
> be used for query inflation, am I right?
Generally, stemming increases recall (proportion of matching relevant docs
among relevant docs in the entire corpus), and decreases precision
(proportion of relevant docs among matching docs).
The standard advice is to use the same analysis pipeline at both index-time
and query-time; in the context of your question, that would mean stemming in
both places.
However, adding stems to a query, especially if you boosted them lower than
the original terms, is probably a good strategy to maximize both precision
and recall. The cost of this approach is two-fold: a larger index than if
you had performed index-time stemming; and increased query-time processing,
hence lower query throughput.
Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Lucene, HTML and Hebrew
Posted by Steven A Rowe <sa...@syr.edu>.
Hi Itamar,
In another thread, you wrote:
> Yesterday I sent an email to this group querying about some
> very important (to me...) features of Lucene. I'm giving it
> another chance before it goes unnoticed or forgotten. If it
> was too long please let me know and I will email a shorter
> list of questions....
I think I have something like a 30-second rule for posts on this list: if I can't figure out what the question is within 30 seconds, I move on. Your post was so verbose that I gave up before I asked myself whether I could help. (Déjà vu - upon re-reading this paragraph, it sounds very much like something Hoss has said on this list...)
Although I answer your original post below, please don't take this as affirmation of your "reminder" approach. In my experience, this strategy is interpreted as badgering, and tends to affect response rate in the opposite direction to that intended.
Short, focused questions will maximize the response rate here (and elsewhere, I suspect). Also, it helps if there is some indication that the questioner has attempted to answer the question for themselves using readily available resources, but failed.
On 01/21/2008 at 2:59 PM, Itamar Syn-Hershko wrote:
> 1) How would Lucene treat the "normal" paragraph when they
> are added that way? Would proximity and frequency data be
> computed between paragraph1 and paragraph2 (last word of
> former with first word of latter)? What about proximity
> data between "h2" paragraph and the "normal" before or
> after it?
Lucene does not store proximity relations between data in different fields, only within individual fields. Similarly, term frequencies are stored per-field, not per-document.
> 2) How would I set the boosts for the headers and footnotes?
> I'd rather have it stored within the index file than have to
> append it to each and every query I will execute, but I'm
> open to suggestions. I'm more interested in performance and
> flexibility.
AFAIK, there is no way currently in Lucene to set index-time per-field boosts - only per-document boosts are supported.
One very coarse-grained boosting trick you could use is to repeat the text of headers, etc., that you want to boost, e.g.:
Doc->add(new Field("h2", "sub-header 1 $$$ sub-header 1", Field::STORE_NO |
Field::INDEX_TOKENIZED);
I included "$$$" as an example of how to break proximity between the first and last terms in the "sub-header 1" text - note, however, that this particular string may not serve this function properly, depending on the analyzer you choose.
Note also that this is an issue elsewhere for you, since each addition of field information is understood by Lucene as contiguous. That is, unless you do something to inhibit it, proximity matches will occur between the last term from one <h2> tag and the first term from the next <h2> tag in the same doc.
> 3) When executing a query against the above-mentioned index,
> how would I execute a set of words as a query (boolean quey
> using list of inflated words) without repeating this set of
> words for each and every field? Any support for something
> like *:word1 OR word2 OR word3 (instead of normal:(word1 OR
> word2 OR word3) AND quote:(word1 OR word2 OR word3) AND
> h1:(word1 OR word2 OR word3) etc...)?
MultiFieldQueryParser may do something like what you want:
<http://lucene.apache.org/java/1_4_3/api/org/apache/lucene/queryParser/MultiFieldQueryParser.html>
> 4) Writing a Hebrew analyzer, I'm considering using a
> StandardAnalyzer and just ammend it so when it recognizes a
> Hebrew word it will call some function that will parse it
> correctly (there are some differences, such as quotes in
> the middle of a word are legitmate, also remove Niqqud).
> If it's a non-Hebrew word it will just continue as usual
> with its default behavior and functions.
> Also, this means I will index ALL words (Hebrew AND English)
> into the same index. The thinking behind this is to allow
> for searches with both Hebrew and English words to be
> performed successfully, taking into account there shouldn't
> be any downsides for indexing two languages within one
> index. I'm aware of the way Lucene stores words (not the
> whole word, but only the part that is different from the
> previous), but I really don't see how bad that's gonna
> be...
Not sure what the question is here - if you mean to ask "What are the impacts of including terms from two languages in a single index?" then my answer is "it depends"...
For languages that share orthographies (e.g. Spanish and French, to a great extent), cognates (i.e. the same term meaning completely different things in the two languages) could cause degraded precision. AFAIK, this is not an issue for Hebrew and English.
The only other issue I can think of is that you will be taking symbols (words) from two completely different meaning-systems and merging them into the same index space. For similar contexts ("should I have separate fields for each unit of information?"), the advice generally given on this list is to put everything into a single field. In short: try the simplest thing first, test, and if the performance is not good enough, then increase the complexity of your solution, test, and iterate until it is. But you probably already knew that :).
> 5) Where should a stemmer be used? As far as I see it, it
> should only be used for query inflation, am I right?
Generally, stemming increases recall (proportion of matching relevant docs among relevant docs in the entire corpus), and decreases precision (proportion of relevant docs among matching docs).
The standard advice is to use the same analysis pipeline at both index-time and query-time; in the context of your question, that would mean stemming in both places.
However, adding stems to a query, especially if you boosted them lower than the original terms, is probably a good strategy to maximize both precision and recall. The cost of this approach is two-fold: a larger index than if you had performed index-time stemming; and increased query-time processing, hence lower query throughput.
Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org