You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Peter Laurinc <la...@felisconsulting.com> on 2005/07/01 14:16:48 UTC
Sentence and Paragraph searching
Hi,
I'm newbie to lucene.
I wan to ask, how to implement search for phrase that must be in
sentence/paragraph.
I did see som examples, that uses term position changing, but I think
that this is not the way, because it breaks classic proximity search.
(if one word is on end and second of begining of next sentence)
Thanks for any help
Peter
Re: Sentence and Paragraph searching
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jul 1, 2005, at 8:16 AM, Peter Laurinc wrote:
> Hi,
>
> I'm newbie to lucene.
> I wan to ask, how to implement search for phrase that must be in
> sentence/paragraph.
> I did see som examples, that uses term position changing, but I think
> that this is not the way, because it breaks classic proximity search.
> (if one word is on end and second of begining of next sentence)
It really depends on your needs. If you never need proximity across
sentence boundaries, then what's the issue? Putting a large gap at
sentence boundaries makes good sense for some needs. Maybe not so
for your situation?
I'm definitely interested in what others have done with this sort of
thing.
At the extreme, if all you wanted was to find sentences and did not
need to query for terms in multiple sentences at one time then you
could index each sentence as a separate Document.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Sentence and Paragraph searching
Posted by Paul Elschot <pa...@xs4all.nl>.
On Friday 01 July 2005 21:13, Dave Kor wrote:
> Quoting Peter Laurinc <la...@felisconsulting.com>:
>
> > Hi,
> >
> > I'm newbie to lucene.
> > I wan to ask, how to implement search for phrase that must be in
> > sentence/paragraph.
> > I did see som examples, that uses term position changing, but I think
> > that this is not the way, because it breaks classic proximity search.
> > (if one word is on end and second of begining of next sentence)
>
> Most NLP toolkits have a sentence and paragraph boundary detector that can be
> used to separate a single document into its constituent sentences and
> paragraphs. The two NLP toolkits I am familiar with are OpenNLP (Open Source)
> and Alias-i's LingPipe (Commercial) libraries. These toolkits can be used in
> several ways to achieve what you want.
>
> If you are ONLY interested in searching for individual sentences, then you can
> use the toolkits to create an index of sentences instead of an index of
> documents.
>
> Alternatively, you can encode the sentence boundaries found by these toolkits
> within the documents you are indexing, for example using special characters or
> as a separate field in Lucene. After every search, do an extra check to ensure
> that Lucene did not match across sentence boundaries.
Or try and use a SpanNotQuery to make sure that the sentence or paragraph
border is contained in matches in the same field.
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Sentence and Paragraph searching
Posted by Dave Kor <s0...@sms.ed.ac.uk>.
Quoting Peter Laurinc <la...@felisconsulting.com>:
> Hi,
>
> I'm newbie to lucene.
> I wan to ask, how to implement search for phrase that must be in
> sentence/paragraph.
> I did see som examples, that uses term position changing, but I think
> that this is not the way, because it breaks classic proximity search.
> (if one word is on end and second of begining of next sentence)
Most NLP toolkits have a sentence and paragraph boundary detector that can be
used to separate a single document into its constituent sentences and
paragraphs. The two NLP toolkits I am familiar with are OpenNLP (Open Source)
and Alias-i's LingPipe (Commercial) libraries. These toolkits can be used in
several ways to achieve what you want.
If you are ONLY interested in searching for individual sentences, then you can
use the toolkits to create an index of sentences instead of an index of
documents.
Alternatively, you can encode the sentence boundaries found by these toolkits
within the documents you are indexing, for example using special characters or
as a separate field in Lucene. After every search, do an extra check to ensure
that Lucene did not match across sentence boundaries.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org