You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Roxana Angheluta <ro...@attentio.com> on 2005/06/21 16:01:41 UTC
index phrases
Dear all,
I am using Lucene for indexing documents.
I would like to include phrases (of a certain maximum length given as a
parameter) in the index. I know this is non-standard for e.g. searching,
where a PhraseQuery can be built which makes use of the terms positions.
However, I am not interested in searching, but rather in using the
indexing terms for some statistics.
What would be an efficient way to do this? Is it possible to build
phrases in a filter after tokenization?
Thanks,
roxana
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: index phrases
Posted by Sebastian Marius Kirsch <sk...@sebastian-kirsch.org>.
On Tue, Jun 21, 2005 at 02:06:31PM -0400, Erik Hatcher wrote:
> A contribution with dependencies is fine, especially Apache ones. We
> can put this code in the contrib area if you'd like to contribute
> it. If so, please create a Bugzilla issue and attach the sources.
Hi Erik,
thanks for the suggestion; I posted the code at
http://issues.apache.org/bugzilla/show_bug.cgi?id=35456
Comments welcome. (And sorry for the messed-up indenting on two of the
source files. Please re-indent.)
Regards, Sebastian
--
Sebastian Kirsch <sk...@sebastian-kirsch.org> [http://www.sebastian-kirsch.org/]
NOTE: New email address! Please update your address book.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: index phrases
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jun 21, 2005, at 12:46 PM, Sebastian Marius Kirsch wrote:
> On Tue, Jun 21, 2005 at 04:01:41PM +0200, Roxana Angheluta wrote:
>
>> I would like to include phrases (of a certain maximum length given
>> as a
>> parameter) in the index. I know this is non-standard for e.g.
>> searching,
>> where a PhraseQuery can be built which makes use of the terms
>> positions.
>> However, I am not interested in searching, but rather in using the
>> indexing terms for some statistics.
>>
>
> I built a filter for exactly this purpose (statistics over word
> combinations); I will send it to you directly. It uses some containers
> from org.apache.commons.collections, so I guess it's not suitable yet
> for inclusion in the lucene distribution.
A contribution with dependencies is fine, especially Apache ones. We
can put this code in the contrib area if you'd like to contribute
it. If so, please create a Bugzilla issue and attach the sources.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Where do I get org.apache.commons.collections package sources?
Posted by karl wettin <ka...@gmail.com>.
On Tue, 2006-09-05 at 02:38 -0700, Venkateshprasanna wrote:
> I saw these classes and want to use them for my implementation as well.
> But I am not getting the source code for the specified package:
> org.apache.commons.collections
http://jakarta.apache.org/commons/collections/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Where do I get org.apache.commons.collections package sources?
Posted by Venkateshprasanna <pr...@yahoo.co.in>.
I saw these classes and want to use them for my implementation as well. But I
am not getting the source code for the specified package:
org.apache.commons.collections
Is there any other way of implementing the same?
Why only classes from that package has to be used?
Regards,
Venkateshprasanna
I built a filter for exactly this purpose (statistics over word
combinations); I will send it to you directly. It uses some containers
from org.apache.commons.collections, so I guess it's not suitable yet
for inclusion in the lucene distribution.
The filter handles position increments > 1 by inserting fillers, so a
phrase "look at my car" will become "look _ _ car" when the words "at"
and "my" are filtered out from the stream, and "car" is assigned a
position increment of 3. But since StopFilter does not currently
produce position increments, you will hardly ever see them. The filter
does not handle position increments == 0. (But will produce them for
ngrams that start at the same position.)
--
View this message in context: http://www.nabble.com/index-phrases-tf90411.html#a6148808
Sent from the Lucene - Java Users forum at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: index phrases
Posted by Sebastian Marius Kirsch <sk...@sebastian-kirsch.org>.
On Tue, Jun 21, 2005 at 04:01:41PM +0200, Roxana Angheluta wrote:
> I would like to include phrases (of a certain maximum length given as a
> parameter) in the index. I know this is non-standard for e.g. searching,
> where a PhraseQuery can be built which makes use of the terms positions.
> However, I am not interested in searching, but rather in using the
> indexing terms for some statistics.
I built a filter for exactly this purpose (statistics over word
combinations); I will send it to you directly. It uses some containers
from org.apache.commons.collections, so I guess it's not suitable yet
for inclusion in the lucene distribution.
The filter handles position increments > 1 by inserting fillers, so a
phrase "look at my car" will become "look _ _ car" when the words "at"
and "my" are filtered out from the stream, and "car" is assigned a
position increment of 3. But since StopFilter does not currently
produce position increments, you will hardly ever see them. The filter
does not handle position increments == 0. (But will produce them for
ngrams that start at the same position.)
I think there is also some stuff for ngram indexing in nutch.
Regards, Sebastian
--
Sebastian Kirsch <sk...@sebastian-kirsch.org> [http://www.sebastian-kirsch.org/]
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: index phrases
Posted by sergiu gordea <gs...@ifit.uni-klu.ac.at>.
Ciau,
Ce mai faci?
Cred ca te-am mai intrebat ce lucrezi, dar am uitat.
In viitorul apropiat s-ar putea sa lucrez si eu putin in directia text
mining
si bineinteles ca am sa refolosesc ce e implementat in lucene.
S-ar putea ca munca noastra sa aiba ceva puncte comune .. te
intereseaza sa
mai schimbam cate o opinie din cand in cand?
Numai bine,
Sergiu
Roxana Angheluta wrote:
>
>>> Dear all,
>>>
>>> I am using Lucene for indexing documents.
>>>
>>> I would like to include phrases (of a certain maximum length given
>>> as a parameter) in the index. I know this is non-standard for e.g.
>>> searching, where a PhraseQuery can be built which makes use of the
>>> terms positions. However, I am not interested in searching, but
>>> rather in using the indexing terms for some statistics.
>>>
>>> What would be an efficient way to do this? Is it possible to build
>>> phrases in a filter after tokenization?
>>
>>
>> Roxana- could you give us a concrete example of what you're wanting
>> to do?
>>
>> A TokenFilter could certainly be used to aggregate multiple terms
>> into a single term that represents a phrase. This would happen
>> during the analysis process, which occurs along with tokenization.
>
> Hi Erik, thanks for the answer.
> I would like to index the following document:
>
> This is a sample document.
>
> something like:
> "this"
> "is"
> "a"
> "sample"
> "document"
> "this is"
> "is a"
> "a sample"
> "this a"
> "is sample"
> "a document"
> "sample document"
> "this is a"
> "is a sample"
> "a sample document"
>
> In this example the maximum length of an n-gram is 3 and the length of
> the moving window accross text is also 3.
> In fact I would like a full analyzer to do the job, i.e. define a
> strategy to filter out/clean spurious n-grams: e.g. remove n-grams
> made out only/partially of stopwords, eliminate just stopwords from
> the n-gram.
>
> Sebastian has kindly provided his code, which does the job.
>
> roxana
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: index phrases
Posted by Roxana Angheluta <ro...@attentio.com>.
>> Dear all,
>>
>> I am using Lucene for indexing documents.
>>
>> I would like to include phrases (of a certain maximum length given
>> as a parameter) in the index. I know this is non-standard for e.g.
>> searching, where a PhraseQuery can be built which makes use of the
>> terms positions. However, I am not interested in searching, but
>> rather in using the indexing terms for some statistics.
>>
>> What would be an efficient way to do this? Is it possible to build
>> phrases in a filter after tokenization?
>
> Roxana- could you give us a concrete example of what you're wanting
> to do?
>
> A TokenFilter could certainly be used to aggregate multiple terms
> into a single term that represents a phrase. This would happen
> during the analysis process, which occurs along with tokenization.
Hi Erik, thanks for the answer.
I would like to index the following document:
This is a sample document.
something like:
"this"
"is"
"a"
"sample"
"document"
"this is"
"is a"
"a sample"
"this a"
"is sample"
"a document"
"sample document"
"this is a"
"is a sample"
"a sample document"
In this example the maximum length of an n-gram is 3 and the length of
the moving window accross text is also 3.
In fact I would like a full analyzer to do the job, i.e. define a
strategy to filter out/clean spurious n-grams: e.g. remove n-grams made
out only/partially of stopwords, eliminate just stopwords from the n-gram.
Sebastian has kindly provided his code, which does the job.
roxana
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: index phrases
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jun 21, 2005, at 10:01 AM, Roxana Angheluta wrote:
> Dear all,
>
> I am using Lucene for indexing documents.
>
> I would like to include phrases (of a certain maximum length given
> as a parameter) in the index. I know this is non-standard for e.g.
> searching, where a PhraseQuery can be built which makes use of the
> terms positions. However, I am not interested in searching, but
> rather in using the indexing terms for some statistics.
>
> What would be an efficient way to do this? Is it possible to build
> phrases in a filter after tokenization?
Roxana- could you give us a concrete example of what you're wanting
to do?
A TokenFilter could certainly be used to aggregate multiple terms
into a single term that represents a phrase. This would happen
during the analysis process, which occurs along with tokenization.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org