You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Bill Taylor <wa...@as-st.com> on 2006/08/29 16:40:04 UTC

Installing a custom tokenizer

I am indexing documents which are filled with government jargon.  As 
one would expect, the standard tokenizer has problems with 
governmenteese.

In particular, the documents use words such as 310N-P-Q as references 
to other documents.  The standard tokenizer breaks this "word" at the 
dashes so that I can find P or Q but not the entire token.

I know how to write a new tokenizer.  I would like hints on how to 
install it and get my indexing system to use it.  I don't want to 
modify the standard .jar file.  What I think I want to do is set up my 
indexing operation to use the WhitespaceTokenizer instead of the normal 
one, but I am unsure how to do this.

I know that the IndexTask has a setAnalyzer method.  The document 
formats are rather complicated and I need special code to isolate the 
text strings which should be indexed.   My file analyzer isolates the 
string I want to index, then does

doc.add(new Field(DocFormatters.CONTENT_FIELD, <string from the file>, 
Field.Store.YES, Field.index.TOKENIZED));

I suspect that my issue is getting the Field constructor to use a 
different tokenizer.  Can anyone help?

Thanks.

Bill Taylor


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Installing a custom tokenizer

Posted by Mark Miller <ma...@gmail.com>.
Bill Taylor wrote:
>
> On Aug 29, 2006, at 1:46 PM, Erick Erickson wrote:
>
>> I'm in a real rush here, so pardon my brevity, but..... one of the
>> constructors for IndexWriter takes an Analyzer as a parameter, which 
>> can be
>> a PerFieldAnalyzerWrapper. That, if I understand your issue, should 
>> fix you
>> right up.
>
> that almost worked.  I can't use a per Field analyzer because I have 
> to process the content fields of all documents.  I built a custom 
> analyzer which extended the Standard Analyzer and replaced the 
> tokenStream method with a new one which used WhitespaceTokenizer 
> instead of StandardTokenizer.  This meant that my document IDs were 
> not split, but I lost the conversion of acronyms such as w.o. to wo 
> and the like
>
> So what I need to do is to make a new Tokenizer based on the 
> StandardTokenizer except that a NUM on line 83 of StandardTokenizer.jj 
> should be
>
> | NUM: (<ALPHANUM> (<P> <ALPHANUM>) +  | <ALPHANUM>) >
>
> so that a serial number need not have a digit in every other segment 
> and a series of letters and digits without special characters such as 
> a dash will be treated as a single word.
>
> Questions:
>
> 1) If I change the .jj file in this way, how to I run javaCC to make a 
> new tokenizer?  The JavaCC documentation says that JavaCC generates a 
> number of output files; I think that I only need the tokenizer code.
>
> 2) I suppose i have to tell the query parser to parse queries in the 
> same way, is that right?
>
> The reason I think so is that Luke says I have words such as w.o. in 
> the index which the query parser can't find.  I suspect I have to use 
> the same Analyzer on both, right?
>
Get JavaCC and run it on StandardTokenizer.jj. This should be as simple 
as typing 'JavaCC StandardTokenizer.jj'...I believe with no output 
folder specified all of the files will be built in the current 
directory. Don't worry about not generating the ones you do not 
need--JavaCC will handle everything for you. If you use Eclipse I 
recommend the JavaCC plug-in. I find it very handy.

Generally you must run the same analyzer that you indexed with on your 
search strings...if the standard analyzer parses oldman-83 to oldman 
while indexing and you use whitespace analyzer while searching then you 
will attempt to find oldman-83 in the index instead of oldman (which was 
what standard analyzer stored).

- Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Straight TF-IDF cosine similarity?

Posted by Jason Polites <ja...@gmail.com>.
Have you looked at the MoreLikeThis class in the similarity package?

On 8/30/06, Winton Davies <wd...@yahoo-inc.com> wrote:
>
> Hi All,
>
> I'm scratching my head - can someone tell me which class implements
> an efficient multiple term TF.IDF Cosine similarity scoring mechanism?
>
> There is clearly the single TermScorer - but I can't find the class
> that would do a bucketed TF.IDF cosine - i.e. fill an accumulator
> with the tf.idf^2 for each of the term posting lists, until
> accumulator is full, and then compute the final score.
>
> I don't need a Boolean Query - at least this seems like overkill.
>
> Cheers,
>   Winton
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Straight TF-IDF cosine similarity?

Posted by Winton Davies <wd...@yahoo-inc.com>.
Hi All,

I'm scratching my head - can someone tell me which class implements 
an efficient multiple term TF.IDF Cosine similarity scoring mechanism?

There is clearly the single TermScorer - but I can't find the class 
that would do a bucketed TF.IDF cosine - i.e. fill an accumulator 
with the tf.idf^2 for each of the term posting lists, until 
accumulator is full, and then compute the final score.

I don't need a Boolean Query - at least this seems like overkill.

Cheers,
  Winton

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Installing a custom tokenizer

Posted by Bill Taylor <wa...@as-st.com>.
On Aug 29, 2006, at 1:46 PM, Erick Erickson wrote:

> I'm in a real rush here, so pardon my brevity, but..... one of the
> constructors for IndexWriter takes an Analyzer as a parameter, which 
> can be
> a PerFieldAnalyzerWrapper. That, if I understand your issue, should 
> fix you
> right up.

that almost worked.  I can't use a per Field analyzer because I have to 
process the content fields of all documents.  I built a custom analyzer 
which extended the Standard Analyzer and replaced the tokenStream 
method with a new one which used WhitespaceTokenizer instead of 
StandardTokenizer.  This meant that my document IDs were not split, but 
I lost the conversion of acronyms such as w.o. to wo and the like

So what I need to do is to make a new Tokenizer based on the 
StandardTokenizer except that a NUM on line 83 of StandardTokenizer.jj 
should be

| NUM: (<ALPHANUM> (<P> <ALPHANUM>) +  | <ALPHANUM>) >

so that a serial number need not have a digit in every other segment 
and a series of letters and digits without special characters such as a 
dash will be treated as a single word.

Questions:

1) If I change the .jj file in this way, how to I run javaCC to make a 
new tokenizer?  The JavaCC documentation says that JavaCC generates a 
number of output files; I think that I only need the tokenizer code.

2) I suppose i have to tell the query parser to parse queries in the 
same way, is that right?

The reason I think so is that Luke says I have words such as w.o. in 
the index which the query parser can't find.  I suspect I have to use 
the same Analyzer on both, right?

> On 8/29/06, Bill Taylor <wa...@as-st.com> wrote:
>>
>> I am indexing documents which are filled with government jargon.  As
>> one would expect, the standard tokenizer has problems with
>> governmenteese.
>>
>> In particular, the documents use words such as 310N-P-Q as references
>> to other documents.  The standard tokenizer breaks this "word" at the
>> dashes so that I can find P or Q but not the entire token.
>>
>> I know how to write a new tokenizer.  I would like hints on how to
>> install it and get my indexing system to use it.  I don't want to
>> modify the standard .jar file.  What I think I want to do is set up my
>> indexing operation to use the WhitespaceTokenizer instead of the 
>> normal
>> one, but I am unsure how to do this.
>>
>> I know that the IndexTask has a setAnalyzer method.  The document
>> formats are rather complicated and I need special code to isolate the
>> text strings which should be indexed.   My file analyzer isolates the
>> string I want to index, then does
>>
>> doc.add(new Field(DocFormatters.CONTENT_FIELD, <string from the file>,
>> Field.Store.YES, Field.index.TOKENIZED));
>>
>> I suspect that my issue is getting the Field constructor to use a
>> different tokenizer.  Can anyone help?
>>
>> Thanks.
>>
>> Bill Taylor
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Installing a custom tokenizer

Posted by Erick Erickson <er...@gmail.com>.
I'm in a real rush here, so pardon my brevity, but..... one of the
constructors for IndexWriter takes an Analyzer as a parameter, which can be
a PerFieldAnalyzerWrapper. That, if I understand your issue, should fix you
right up.

Same kind of thing for a Query.

Erick

On 8/29/06, Bill Taylor <wa...@as-st.com> wrote:
>
> I am indexing documents which are filled with government jargon.  As
> one would expect, the standard tokenizer has problems with
> governmenteese.
>
> In particular, the documents use words such as 310N-P-Q as references
> to other documents.  The standard tokenizer breaks this "word" at the
> dashes so that I can find P or Q but not the entire token.
>
> I know how to write a new tokenizer.  I would like hints on how to
> install it and get my indexing system to use it.  I don't want to
> modify the standard .jar file.  What I think I want to do is set up my
> indexing operation to use the WhitespaceTokenizer instead of the normal
> one, but I am unsure how to do this.
>
> I know that the IndexTask has a setAnalyzer method.  The document
> formats are rather complicated and I need special code to isolate the
> text strings which should be indexed.   My file analyzer isolates the
> string I want to index, then does
>
> doc.add(new Field(DocFormatters.CONTENT_FIELD, <string from the file>,
> Field.Store.YES, Field.index.TOKENIZED));
>
> I suspect that my issue is getting the Field constructor to use a
> different tokenizer.  Can anyone help?
>
> Thanks.
>
> Bill Taylor
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>