You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Bill Taylor <wa...@as-st.com> on 2006/08/29 16:40:04 UTC
Installing a custom tokenizer
I am indexing documents which are filled with government jargon. As
one would expect, the standard tokenizer has problems with
governmenteese.
In particular, the documents use words such as 310N-P-Q as references
to other documents. The standard tokenizer breaks this "word" at the
dashes so that I can find P or Q but not the entire token.
I know how to write a new tokenizer. I would like hints on how to
install it and get my indexing system to use it. I don't want to
modify the standard .jar file. What I think I want to do is set up my
indexing operation to use the WhitespaceTokenizer instead of the normal
one, but I am unsure how to do this.
I know that the IndexTask has a setAnalyzer method. The document
formats are rather complicated and I need special code to isolate the
text strings which should be indexed. My file analyzer isolates the
string I want to index, then does
doc.add(new Field(DocFormatters.CONTENT_FIELD, <string from the file>,
Field.Store.YES, Field.index.TOKENIZED));
I suspect that my issue is getting the Field constructor to use a
different tokenizer. Can anyone help?
Thanks.
Bill Taylor
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Installing a custom tokenizer
Posted by Mark Miller <ma...@gmail.com>.
Bill Taylor wrote:
>
> On Aug 29, 2006, at 1:46 PM, Erick Erickson wrote:
>
>> I'm in a real rush here, so pardon my brevity, but..... one of the
>> constructors for IndexWriter takes an Analyzer as a parameter, which
>> can be
>> a PerFieldAnalyzerWrapper. That, if I understand your issue, should
>> fix you
>> right up.
>
> that almost worked. I can't use a per Field analyzer because I have
> to process the content fields of all documents. I built a custom
> analyzer which extended the Standard Analyzer and replaced the
> tokenStream method with a new one which used WhitespaceTokenizer
> instead of StandardTokenizer. This meant that my document IDs were
> not split, but I lost the conversion of acronyms such as w.o. to wo
> and the like
>
> So what I need to do is to make a new Tokenizer based on the
> StandardTokenizer except that a NUM on line 83 of StandardTokenizer.jj
> should be
>
> | NUM: (<ALPHANUM> (<P> <ALPHANUM>) + | <ALPHANUM>) >
>
> so that a serial number need not have a digit in every other segment
> and a series of letters and digits without special characters such as
> a dash will be treated as a single word.
>
> Questions:
>
> 1) If I change the .jj file in this way, how to I run javaCC to make a
> new tokenizer? The JavaCC documentation says that JavaCC generates a
> number of output files; I think that I only need the tokenizer code.
>
> 2) I suppose i have to tell the query parser to parse queries in the
> same way, is that right?
>
> The reason I think so is that Luke says I have words such as w.o. in
> the index which the query parser can't find. I suspect I have to use
> the same Analyzer on both, right?
>
Get JavaCC and run it on StandardTokenizer.jj. This should be as simple
as typing 'JavaCC StandardTokenizer.jj'...I believe with no output
folder specified all of the files will be built in the current
directory. Don't worry about not generating the ones you do not
need--JavaCC will handle everything for you. If you use Eclipse I
recommend the JavaCC plug-in. I find it very handy.
Generally you must run the same analyzer that you indexed with on your
search strings...if the standard analyzer parses oldman-83 to oldman
while indexing and you use whitespace analyzer while searching then you
will attempt to find oldman-83 in the index instead of oldman (which was
what standard analyzer stored).
- Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Straight TF-IDF cosine similarity?
Posted by Jason Polites <ja...@gmail.com>.
Have you looked at the MoreLikeThis class in the similarity package?
On 8/30/06, Winton Davies <wd...@yahoo-inc.com> wrote:
>
> Hi All,
>
> I'm scratching my head - can someone tell me which class implements
> an efficient multiple term TF.IDF Cosine similarity scoring mechanism?
>
> There is clearly the single TermScorer - but I can't find the class
> that would do a bucketed TF.IDF cosine - i.e. fill an accumulator
> with the tf.idf^2 for each of the term posting lists, until
> accumulator is full, and then compute the final score.
>
> I don't need a Boolean Query - at least this seems like overkill.
>
> Cheers,
> Winton
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Straight TF-IDF cosine similarity?
Posted by Winton Davies <wd...@yahoo-inc.com>.
Hi All,
I'm scratching my head - can someone tell me which class implements
an efficient multiple term TF.IDF Cosine similarity scoring mechanism?
There is clearly the single TermScorer - but I can't find the class
that would do a bucketed TF.IDF cosine - i.e. fill an accumulator
with the tf.idf^2 for each of the term posting lists, until
accumulator is full, and then compute the final score.
I don't need a Boolean Query - at least this seems like overkill.
Cheers,
Winton
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Installing a custom tokenizer
Posted by Bill Taylor <wa...@as-st.com>.
On Aug 29, 2006, at 1:46 PM, Erick Erickson wrote:
> I'm in a real rush here, so pardon my brevity, but..... one of the
> constructors for IndexWriter takes an Analyzer as a parameter, which
> can be
> a PerFieldAnalyzerWrapper. That, if I understand your issue, should
> fix you
> right up.
that almost worked. I can't use a per Field analyzer because I have to
process the content fields of all documents. I built a custom analyzer
which extended the Standard Analyzer and replaced the tokenStream
method with a new one which used WhitespaceTokenizer instead of
StandardTokenizer. This meant that my document IDs were not split, but
I lost the conversion of acronyms such as w.o. to wo and the like
So what I need to do is to make a new Tokenizer based on the
StandardTokenizer except that a NUM on line 83 of StandardTokenizer.jj
should be
| NUM: (<ALPHANUM> (<P> <ALPHANUM>) + | <ALPHANUM>) >
so that a serial number need not have a digit in every other segment
and a series of letters and digits without special characters such as a
dash will be treated as a single word.
Questions:
1) If I change the .jj file in this way, how to I run javaCC to make a
new tokenizer? The JavaCC documentation says that JavaCC generates a
number of output files; I think that I only need the tokenizer code.
2) I suppose i have to tell the query parser to parse queries in the
same way, is that right?
The reason I think so is that Luke says I have words such as w.o. in
the index which the query parser can't find. I suspect I have to use
the same Analyzer on both, right?
> On 8/29/06, Bill Taylor <wa...@as-st.com> wrote:
>>
>> I am indexing documents which are filled with government jargon. As
>> one would expect, the standard tokenizer has problems with
>> governmenteese.
>>
>> In particular, the documents use words such as 310N-P-Q as references
>> to other documents. The standard tokenizer breaks this "word" at the
>> dashes so that I can find P or Q but not the entire token.
>>
>> I know how to write a new tokenizer. I would like hints on how to
>> install it and get my indexing system to use it. I don't want to
>> modify the standard .jar file. What I think I want to do is set up my
>> indexing operation to use the WhitespaceTokenizer instead of the
>> normal
>> one, but I am unsure how to do this.
>>
>> I know that the IndexTask has a setAnalyzer method. The document
>> formats are rather complicated and I need special code to isolate the
>> text strings which should be indexed. My file analyzer isolates the
>> string I want to index, then does
>>
>> doc.add(new Field(DocFormatters.CONTENT_FIELD, <string from the file>,
>> Field.Store.YES, Field.index.TOKENIZED));
>>
>> I suspect that my issue is getting the Field constructor to use a
>> different tokenizer. Can anyone help?
>>
>> Thanks.
>>
>> Bill Taylor
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Installing a custom tokenizer
Posted by Erick Erickson <er...@gmail.com>.
I'm in a real rush here, so pardon my brevity, but..... one of the
constructors for IndexWriter takes an Analyzer as a parameter, which can be
a PerFieldAnalyzerWrapper. That, if I understand your issue, should fix you
right up.
Same kind of thing for a Query.
Erick
On 8/29/06, Bill Taylor <wa...@as-st.com> wrote:
>
> I am indexing documents which are filled with government jargon. As
> one would expect, the standard tokenizer has problems with
> governmenteese.
>
> In particular, the documents use words such as 310N-P-Q as references
> to other documents. The standard tokenizer breaks this "word" at the
> dashes so that I can find P or Q but not the entire token.
>
> I know how to write a new tokenizer. I would like hints on how to
> install it and get my indexing system to use it. I don't want to
> modify the standard .jar file. What I think I want to do is set up my
> indexing operation to use the WhitespaceTokenizer instead of the normal
> one, but I am unsure how to do this.
>
> I know that the IndexTask has a setAnalyzer method. The document
> formats are rather complicated and I need special code to isolate the
> text strings which should be indexed. My file analyzer isolates the
> string I want to index, then does
>
> doc.add(new Field(DocFormatters.CONTENT_FIELD, <string from the file>,
> Field.Store.YES, Field.index.TOKENIZED));
>
> I suspect that my issue is getting the Field constructor to use a
> different tokenizer. Can anyone help?
>
> Thanks.
>
> Bill Taylor
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>