You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Chris Tomlinson <ch...@gmail.com> on 2014/04/23 18:36:15 UTC
What is the proper use of stop words in Lucene?
Hello,
I've written several times now on the list with this question / problem and no one has yet replied so I don't know if the question is too wrong-headed or if there is simply no one reading the list that can comment on the question.
The question that I'm trying to get answered is what is the correct way of ignoring stop word gaps in Lucene 4.4+?
While we are using Lucene 4.4 embedded in eXist-db (exist-db.org), I think the question is a proper Lucene question and really has nothing to do with the fact that we're using it in an embedded manner.
The problem to be solved is how to ignore stop word gaps in queries - without the user having to indicate where such gaps might occur at query time.
Since Lucene 4.4 the FilteringTokenFilter.setEnablePositionIncrements(false) is not available. None of the resources such as the "Lucene in Action" and so on explain how to use Lucene to get the desired effect now that 4.4+ has removed the previous approach.
Prior to Lucene 4.4 it was possible to setEnablePositionIncrements(false) so that during indexing and querying the number and position of stop word gaps would be ignored (as mentioned on pp 138-139 of "Lucene in Action").
This meant that a document with a phrase such as:
blue is the sky
with stop words "is" and "the" would be selected by the query:
blue sky
This is what we want to achieve.
Why? We are working with Tibetan and elisions are not uncommon so that, e.g.:
rin po che
on some occasions might be shortened to
rin che
and we would like to have a query of
rin po che
or
rin che
find all occurrences of
rin po che
and
rin che
without having the user have to mark where elisions might occur.
The org.apache.lucene.queryparser.flexible.standard.CommonQueryParserConfiguration provides a setEnablePositionIncrements but that does not seem to work to allow for the above desired query behavior that was possible prior to Lucene 4.4.
What is the proper way to ignore stop word gaps?
Thank you,
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: What is the proper use of stop words in Lucene?
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi,
I think you final goal is not full related to stop word elimination.
I would use synonyms instead of setEnablePositionIncrements.
Alternatively,
Assuming that you have list of stop words, you may simulate previous behaviorsetEnablePositionIncrements(false)
via org.apache.lucene.analysis.MappingCharFilter
" the " => " "
" a " => " "
etc. However you may have to expand your list. e.g. The the THE etc.
Ahmet
On Wednesday, April 23, 2014 7:36 PM, Chris Tomlinson <ch...@gmail.com> wrote:
Hello,
I've written several times now on the list with this question / problem and no one has yet replied so I don't know if the question is too wrong-headed or if there is simply no one reading the list that can comment on the question.
The question that I'm trying to get answered is what is the correct way of ignoring stop word gaps in Lucene 4.4+?
While we are using Lucene 4.4 embedded in eXist-db (exist-db.org), I think the question is a proper Lucene question and really has nothing to do with the fact that we're using it in an embedded manner.
The problem to be solved is how to ignore stop word gaps in queries - without the user having to indicate where such gaps might occur at query time.
Since Lucene 4.4 the FilteringTokenFilter.setEnablePositionIncrements(false) is not available. None of the resources such as the "Lucene in Action" and so on explain how to use Lucene to get the desired effect now that 4.4+ has removed the previous approach.
Prior to Lucene 4.4 it was possible to setEnablePositionIncrements(false) so that during indexing and querying the number and position of stop word gaps would be ignored (as mentioned on pp 138-139 of "Lucene in Action").
This meant that a document with a phrase such as:
blue is the sky
with stop words "is" and "the" would be selected by the query:
blue sky
This is what we want to achieve.
Why? We are working with Tibetan and elisions are not uncommon so that, e.g.:
rin po che
on some occasions might be shortened to
rin che
and we would like to have a query of
rin po che
or
rin che
find all occurrences of
rin po che
and
rin che
without having the user have to mark where elisions might occur.
The org.apache.lucene.queryparser.flexible.standard.CommonQueryParserConfiguration provides a setEnablePositionIncrements but that does not seem to work to allow for the above desired query behavior that was possible prior to Lucene 4.4.
What is the proper way to ignore stop word gaps?
Thank you,
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: What is the proper use of stop words in Lucene?
Posted by Uwe Schindler <uw...@thetaphi.de>.
> Hello Uwe,
>
> Thank you for the reply. I see that there is a version check for the
> use of setEnablePositionIncrements(false); and, I think I may be able
> to use an earlier api with the eXist-db embedding of Lucene 4.4 to
> avoid the version check.
Hi,
you don't need an older version of the Lucene library. It is enough to pass the constant, also with Lucene 4.7 or 4.8 (release in a moment):
sf = new StopFilter(Version.LUCENE_43, ...); sf. setEnablePositionIncrements (false);
The version constant is exactly to use some components that changed in an incompatible way still in later versions, and preserve index/behavior compatibility.
About stop words: What you are doing, is not really "stop words". The main reason for stop words is the following:
- Stop words are in almost every document, so it makes no sense to query for them.
- The only relevant information behind the stop word is "there was a word at this position that"
If the second item would not be taken care, this information would get lost, too.
If every document really contains a specific stop word (which is almost always the case), there must be no difference between a phrase query with mentioned stop word, using an index with all stop words indexed and one with stop words left out. This can only be done, if the stop word reserves a position.
What you intend to do is not a "stopword" use case. You want to "ignore" some words - Lucene has no support for this, because in native language processing this makes no sense. One way to do this is to:
a) write your own TokenFilter, violating the TokenStream contracts
b) use the Backwards compatibility layer with matchVersion=LUCENE_43
c) maybe remove the words before tokenizing (e.g. MappingCharFilter, mapping the "ignore words" to empty string)
Uwe
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: What is the proper use of stop words in Lucene?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Here is the ElisionFilter of Lucene:
https://lucene.apache.org/core/4_8_0/analyzers-common/org/apache/lucene/analysis/util/ElisionFilter.html
This one only works with apostrophe elisions (' and U+2019), so maybe does not apply for Tibetan. But it should inspire you.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, April 28, 2014 10:36 PM
> To: java-user@lucene.apache.org
> Cc: 'Chris Tomlinson'
> Subject: RE: What is the proper use of stop words in Lucene?
>
> Hi,
>
> > > What you intend to do is not a "stopword" use case. You want to "ignore"
> > some words - Lucene has no support for this, because in native
> > language processing this makes no sense.
> >
> > Thank you for the information. I was unaware that ignoring some words
> > "makes no sense". I thought I gave a reasonable example of exactly
> > this situation in the native processing of Tibetan. Perhaps I am still
> > not understanding.
>
> Elisions are a bit different than stopwords (although I don't know about them
> in Tibet language). The Tokenizer should *not* split Elisions from the terms
> (initially the term is the full word including the elision). In most languages
> those are separated by (for example) an apostrophe (e.g. French: le + arbre
> → l’arbre). The Tokenizer would keep those parts together (l’arbre). A later
> TokenFilter would then edit the token and remove the elision (if needed):
> arbre. This is how the French Analyzer in Lucene works.
>
> Lucene currently does not have Tibetanian Analyzer, so you have to make
> your own one (I think this is what you tried to do). You should carefully
> choose the Tokenizer and add something like an TibetanElisionFilter that
> removes the not wanted parts from the tokens.
>
> Uwe
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: What is the proper use of stop words in Lucene?
Posted by Chris Tomlinson <ch...@gmail.com>.
On Apr 28, 2014, at 3:36 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
>>> What you intend to do is not a "stopword" use case. You want to "ignore"
>> some words - Lucene has no support for this, because in native language
>> processing this makes no sense.
>>
>> Thank you for the information. I was unaware that ignoring some words
>> "makes no sense". I thought I gave a reasonable example of exactly this
>> situation in the native processing of Tibetan. Perhaps I am still not
>> understanding.
>
> Elisions are a bit different than stopwords (although I don't know about them in Tibet language). The Tokenizer should *not* split Elisions from the terms (initially the term is the full word including the elision). In most languages those are separated by (for example) an apostrophe (e.g. French: le + arbre → l’arbre). The Tokenizer would keep those parts together (l’arbre). A later TokenFilter would then edit the token and remove the elision (if needed): arbre. This is how the French Analyzer in Lucene works.
Tibetan has no markers for elisions. They can be quite idiosyncratic to a school or tradition. It seems that the most flexible approach is to ignore stop words and their positions - however, incorrect that may be; otherwise, one gets into ever more complex analysis that may not yield cost-effective results. We'll work with the pre-4.4 setEnablePositionIncrements(false) approach for now.
Tibetan has no sentence or phrase markers per sé. There are no word boundary markers. Essentially, Tibetan is a sequence of syllables with occasional markers indicating some break in a thought or what not.
> Lucene currently does not have Tibetanian Analyzer, so you have to make your own one (I think this is what you tried to do). You should carefully choose the Tokenizer and add something like an TibetanElisionFilter that removes the not wanted parts from the tokens.
We have developed a pair of analyzers and associated filters and tokenizers for both Tibetan Unicode and the Extended Wylie transliteration system. If there is interest we will be happy to donate this work to Apache Lucene. This includes paying attention to the myriad punctuation characters, stemming and so on.
Thank you,
Chris
>
> Uwe
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: What is the proper use of stop words in Lucene?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
> > What you intend to do is not a "stopword" use case. You want to "ignore"
> some words - Lucene has no support for this, because in native language
> processing this makes no sense.
>
> Thank you for the information. I was unaware that ignoring some words
> "makes no sense". I thought I gave a reasonable example of exactly this
> situation in the native processing of Tibetan. Perhaps I am still not
> understanding.
Elisions are a bit different than stopwords (although I don't know about them in Tibet language). The Tokenizer should *not* split Elisions from the terms (initially the term is the full word including the elision). In most languages those are separated by (for example) an apostrophe (e.g. French: le + arbre → l’arbre). The Tokenizer would keep those parts together (l’arbre). A later TokenFilter would then edit the token and remove the elision (if needed): arbre. This is how the French Analyzer in Lucene works.
Lucene currently does not have Tibetanian Analyzer, so you have to make your own one (I think this is what you tried to do). You should carefully choose the Tokenizer and add something like an TibetanElisionFilter that removes the not wanted parts from the tokens.
Uwe
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: What is the proper use of stop words in Lucene?
Posted by Chris Tomlinson <ch...@gmail.com>.
Hi,
On Apr 28, 2014, at 11:45 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>> Hello Uwe,
>>
>> Thank you for the reply. I see that there is a version check for the use of
>> setEnablePositionIncrements(false); and, I think I may be able to use an
>> earlier api with the eXist-db embedding of Lucene 4.4 to avoid the version
>> check.
>
> Hi,
>
> you don't need an older version of the Lucene library. It is enough to pass the constant, also with Lucene 4.7 or 4.8 (release in a moment):
> sf = new StopFilter(Version.LUCENE_43, ...);
> sf. setEnablePositionIncrements (false);
>
> The version constant is exactly to use some components that changed in an incompatible way still in later versions, and preserve index/behavior compatibility.
Thank you for the explanation.
> About stop words: What you are doing, is not really "stop words". The main reason for stop words is the following:
> - Stop words are in almost every document, so it makes no sense to query for them.
This was my understanding.
> - The only relevant information behind the stop word is "there was a word at this position that"
I didn't realize that this was a "necessary" aspect. I can certainly understand that it may be relevant in some (most) cases and it makes sense to me that it would appropriate to always preserve the information in indexing. I was looking for a solution that would essentially work at query time and had initially thought that the CommonQueryParserConfiguration#setEnablePositionIncrements() was intended to work this way but it does not.
> If the second item would not be taken care, this information would get lost, too.
>
> If every document really contains a specific stop word (which is almost always the case), there must be no difference between a phrase query with mentioned stop word, using an index with all stop words indexed and one with stop words left out. This can only be done, if the stop word reserves a position.
>
> What you intend to do is not a "stopword" use case. You want to "ignore" some words - Lucene has no support for this, because in native language processing this makes no sense.
Thank you for the information. I was unaware that ignoring some words "makes no sense". I thought I gave a reasonable example of exactly this situation in the native processing of Tibetan. Perhaps I am still not understanding.
> One way to do this is to:
> a) write your own TokenFilter, violating the TokenStream contracts
> b) use the Backwards compatibility layer with matchVersion=LUCENE_43
> c) maybe remove the words before tokenizing (e.g. MappingCharFilter, mapping the "ignore words" to empty string)
Thank you for these useful approaches to solving the use case.
ciao,
Chris
>
> Uwe
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: What is the proper use of stop words in Lucene?
Posted by Chris Tomlinson <ch...@gmail.com>.
Hello Uwe,
Thank you for the reply. I see that there is a version check for the use of setEnablePositionIncrements(false); and, I think I may be able to use an earlier api with the eXist-db embedding of Lucene 4.4 to avoid the version check.
However, my question was intended to improve my understanding of how to properly use stop words and/or how to properly achieve the use case that I outlined.
My naive understanding of the purpose of stop words is:
to remove from indexing words that are not helpful in discriminating or selecting documents since they occur so frequently.
The use case that I intended to illustrate is meant to ignore the occurrence or non-occurrence of stop words in a query w.r.t. selection of documents during search.
As I understand the situation currently, occurrences of stop words in a query phrase are replaced by "?"s to indicate the presence of an otherwise unspecified word in the query. So the phrase:
blue is the moon
with "is" and "the" as stop words, would be indexed effectively as:
blue ? ? moon
and the query phrase:
blue was a moon
would be treated as:
blue ? ? moon
and would retrieve a document containing:
blue is the moon
But in the use case that I presented we really want the query:
blue moon
to also select the document without the user having to indicate the possible presence of stop words or not.
So my question is:
How is one supposed to achieve this use case in Lucene 4.4+?
Thank you,
Chris
On Apr 24, 2014, at 5:52 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
> You can still change the setting on the TokenFilter after creating it: StopFilter#setEnablePositionIncrements(false) - this method was *not* removed!
> This fails only is you pass matchVersion>=Version.LUCENE_44. Just use an older matchVersion parameter to the constructor and you can still enable this broken behavior (for backwards compatibility).
>
> This is no longer officially supported, but can be a workaround. To me it looks like you misunderstood stopwords.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Tincu Gabriel [mailto:tincu.gabriel@gmail.com]
>> Sent: Thursday, April 24, 2014 12:27 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: What is the proper use of stop words in Lucene?
>>
>> Hi there,
>> The StopFilterFactory can be used to produce StopFilters with the desired
>> stop-words inside of it . As a constructor argument it takes a
>> Map<String,String> and one of the valid keys you can pass inside of that is
>> "enablePositionIncrements" . If you don't pass that in then it defaults to true.
>> Is this what you were looking for?
>>
>>
>> On Wed, Apr 23, 2014 at 12:36 PM, Chris Tomlinson <
>> chris.j.tomlinson@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I've written several times now on the list with this question /
>>> problem and no one has yet replied so I don't know if the question is
>>> too wrong-headed or if there is simply no one reading the list that
>>> can comment on the question.
>>>
>>> The question that I'm trying to get answered is what is the correct
>>> way of ignoring stop word gaps in Lucene 4.4+?
>>>
>>> While we are using Lucene 4.4 embedded in eXist-db (exist-db.org), I
>>> think the question is a proper Lucene question and really has nothing
>>> to do with the fact that we're using it in an embedded manner.
>>>
>>> The problem to be solved is how to ignore stop word gaps in queries -
>>> without the user having to indicate where such gaps might occur at
>>> query time.
>>>
>>> Since Lucene 4.4 the
>>> FilteringTokenFilter.setEnablePositionIncrements(false) is not available.
>>> None of the resources such as the "Lucene in Action" and so on explain
>>> how to use Lucene to get the desired effect now that 4.4+ has removed
>>> the previous approach.
>>>
>>> Prior to Lucene 4.4 it was possible to
>>> setEnablePositionIncrements(false)
>>> so that during indexing and querying the number and position of stop
>>> word gaps would be ignored (as mentioned on pp 138-139 of "Lucene in
>> Action").
>>>
>>> This meant that a document with a phrase such as:
>>>
>>> blue is the sky
>>>
>>> with stop words "is" and "the" would be selected by the query:
>>>
>>> blue sky
>>>
>>> This is what we want to achieve.
>>>
>>> Why? We are working with Tibetan and elisions are not uncommon so
>>> that,
>>> e.g.:
>>>
>>> rin po che
>>>
>>> on some occasions might be shortened to
>>>
>>> rin che
>>>
>>> and we would like to have a query of
>>>
>>> rin po che
>>>
>>> or
>>>
>>> rin che
>>>
>>> find all occurrences of
>>>
>>> rin po che
>>>
>>> and
>>>
>>> rin che
>>>
>>> without having the user have to mark where elisions might occur.
>>>
>>> The
>>>
>> org.apache.lucene.queryparser.flexible.standard.CommonQueryParserConfi
>>> guration provides a setEnablePositionIncrements but that does not seem
>>> to work to allow for the above desired query behavior that was
>>> possible prior to Lucene 4.4.
>>>
>>> What is the proper way to ignore stop word gaps?
>>>
>>> Thank you,
>>> Chris
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: What is the proper use of stop words in Lucene?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
You can still change the setting on the TokenFilter after creating it: StopFilter#setEnablePositionIncrements(false) - this method was *not* removed!
This fails only is you pass matchVersion>=Version.LUCENE_44. Just use an older matchVersion parameter to the constructor and you can still enable this broken behavior (for backwards compatibility).
This is no longer officially supported, but can be a workaround. To me it looks like you misunderstood stopwords.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Tincu Gabriel [mailto:tincu.gabriel@gmail.com]
> Sent: Thursday, April 24, 2014 12:27 PM
> To: java-user@lucene.apache.org
> Subject: Re: What is the proper use of stop words in Lucene?
>
> Hi there,
> The StopFilterFactory can be used to produce StopFilters with the desired
> stop-words inside of it . As a constructor argument it takes a
> Map<String,String> and one of the valid keys you can pass inside of that is
> "enablePositionIncrements" . If you don't pass that in then it defaults to true.
> Is this what you were looking for?
>
>
> On Wed, Apr 23, 2014 at 12:36 PM, Chris Tomlinson <
> chris.j.tomlinson@gmail.com> wrote:
>
> > Hello,
> >
> > I've written several times now on the list with this question /
> > problem and no one has yet replied so I don't know if the question is
> > too wrong-headed or if there is simply no one reading the list that
> > can comment on the question.
> >
> > The question that I'm trying to get answered is what is the correct
> > way of ignoring stop word gaps in Lucene 4.4+?
> >
> > While we are using Lucene 4.4 embedded in eXist-db (exist-db.org), I
> > think the question is a proper Lucene question and really has nothing
> > to do with the fact that we're using it in an embedded manner.
> >
> > The problem to be solved is how to ignore stop word gaps in queries -
> > without the user having to indicate where such gaps might occur at
> > query time.
> >
> > Since Lucene 4.4 the
> > FilteringTokenFilter.setEnablePositionIncrements(false) is not available.
> > None of the resources such as the "Lucene in Action" and so on explain
> > how to use Lucene to get the desired effect now that 4.4+ has removed
> > the previous approach.
> >
> > Prior to Lucene 4.4 it was possible to
> > setEnablePositionIncrements(false)
> > so that during indexing and querying the number and position of stop
> > word gaps would be ignored (as mentioned on pp 138-139 of "Lucene in
> Action").
> >
> > This meant that a document with a phrase such as:
> >
> > blue is the sky
> >
> > with stop words "is" and "the" would be selected by the query:
> >
> > blue sky
> >
> > This is what we want to achieve.
> >
> > Why? We are working with Tibetan and elisions are not uncommon so
> > that,
> > e.g.:
> >
> > rin po che
> >
> > on some occasions might be shortened to
> >
> > rin che
> >
> > and we would like to have a query of
> >
> > rin po che
> >
> > or
> >
> > rin che
> >
> > find all occurrences of
> >
> > rin po che
> >
> > and
> >
> > rin che
> >
> > without having the user have to mark where elisions might occur.
> >
> > The
> >
> org.apache.lucene.queryparser.flexible.standard.CommonQueryParserConfi
> > guration provides a setEnablePositionIncrements but that does not seem
> > to work to allow for the above desired query behavior that was
> > possible prior to Lucene 4.4.
> >
> > What is the proper way to ignore stop word gaps?
> >
> > Thank you,
> > Chris
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: What is the proper use of stop words in Lucene?
Posted by Tincu Gabriel <ti...@gmail.com>.
Hi there,
The StopFilterFactory can be used to produce StopFilters with the desired
stop-words inside of it . As a constructor argument it takes a
Map<String,String> and one of the valid keys you can pass inside of that is
"enablePositionIncrements" . If you don't pass that in then it defaults to
true. Is this what you were looking for?
On Wed, Apr 23, 2014 at 12:36 PM, Chris Tomlinson <
chris.j.tomlinson@gmail.com> wrote:
> Hello,
>
> I've written several times now on the list with this question / problem
> and no one has yet replied so I don't know if the question is too
> wrong-headed or if there is simply no one reading the list that can comment
> on the question.
>
> The question that I'm trying to get answered is what is the correct way of
> ignoring stop word gaps in Lucene 4.4+?
>
> While we are using Lucene 4.4 embedded in eXist-db (exist-db.org), I
> think the question is a proper Lucene question and really has nothing to do
> with the fact that we're using it in an embedded manner.
>
> The problem to be solved is how to ignore stop word gaps in queries -
> without the user having to indicate where such gaps might occur at query
> time.
>
> Since Lucene 4.4 the
> FilteringTokenFilter.setEnablePositionIncrements(false) is not available.
> None of the resources such as the "Lucene in Action" and so on explain how
> to use Lucene to get the desired effect now that 4.4+ has removed the
> previous approach.
>
> Prior to Lucene 4.4 it was possible to setEnablePositionIncrements(false)
> so that during indexing and querying the number and position of stop word
> gaps would be ignored (as mentioned on pp 138-139 of "Lucene in Action").
>
> This meant that a document with a phrase such as:
>
> blue is the sky
>
> with stop words "is" and "the" would be selected by the query:
>
> blue sky
>
> This is what we want to achieve.
>
> Why? We are working with Tibetan and elisions are not uncommon so that,
> e.g.:
>
> rin po che
>
> on some occasions might be shortened to
>
> rin che
>
> and we would like to have a query of
>
> rin po che
>
> or
>
> rin che
>
> find all occurrences of
>
> rin po che
>
> and
>
> rin che
>
> without having the user have to mark where elisions might occur.
>
> The
> org.apache.lucene.queryparser.flexible.standard.CommonQueryParserConfiguration
> provides a setEnablePositionIncrements but that does not seem to work to
> allow for the above desired query behavior that was possible prior to
> Lucene 4.4.
>
> What is the proper way to ignore stop word gaps?
>
> Thank you,
> Chris
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>