You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Terry Steichen <te...@net-frame.com> on 2002/12/26 22:38:07 UTC
Incomprehensible (to me) tokenizing behavior
I tested StandardAnalyzer (which uses StandardTokenizer) by inputing the a set of strings which produced the following results:
"aa/bb/cc/dd" was tokenized into 4 terms: aa, bb, cc, dd
"aa/bb/cc/d1" was tokenized into 3 terms: aa, bb, cc/d1
"aa/bb/c1/dd" was tokenized into 2 terms: aa, bb/c1/dd
"aa/b1/cc/dd" was tokenized into 2 terms: aa/b1/cc, dd
"a1/bb/cc/dd" was tokenized into 3 terms: a1/bb, cc, dd
It seems that if the input string includes a numerical value, any first preceeding and/or next following slash ('/') is treated as a character. Otherwise the slash is apparently treated as a token separator.
I'm lost. Assuming this is not a bug, could somebody explain the rhyme and reason to this tokenizing logic?
Regards,
Terry
PS: Using 1.3-dev 1
Re: Incomprehensible (to me) tokenizing behavior
Posted by Doug Cutting <cu...@lucene.com>.
Terry Steichen wrote:
> PS: Is this kind of thing (and more importantly, any other similar
> design issues) documented any place?
This one is described in the source code, with the comment:
// floating point, serial, model numbers, ip addresses, etc.
// every other segment must have at least one digit
> PSS: What is the simplest way to alter this behavior to one that parses the
> same regardless of the presence or absence of numeric characters?
According to:
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/analysis/standard/StandardTokenizer.html
"Many applications have specific tokenizer needs. If this tokenizer does
not suit your application, please consider copying this source code
directory to your project and maintaining your own grammar-based tokenizer."
You need to copy StandardTokenizer.jj, change its package statement, add
some import statements, add a JavaCC task to your build.xml, and,
finally, modify the clause following the above comment.
Doug
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Incomprehensible (to me) tokenizing behavior
Posted by Terry Steichen <te...@net-frame.com>.
Doug,
Aha. I feel better knowing that I haven't lost my mind, now that I know
what you were trying to do.
As to a suggestion, I would only venture to say that, in its present form,
this results in confusing behavior (as noted in my original message).
Whether that drawback is outweighed by the benefits you mention, I could not
say in a general sense (although from my personal use, it is not).
Regards,
Terry
PS: Is this kind of thing (and more importantly, any other similar design
issues) documented any place?
PSS: What is the simplest way to alter this behavior to one that parses the
same regardless of the presence or absence of numeric characters?
----- Original Message -----
From: "Doug Cutting" <cu...@lucene.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Monday, December 30, 2002 1:42 PM
Subject: Re: Incomprehensible (to me) tokenizing behavior
> Terry Steichen wrote:
> > I tested StandardAnalyzer (which uses StandardTokenizer) by inputing the
a set of strings which produced the following results:
> >
> > "aa/bb/cc/dd" was tokenized into 4 terms: aa, bb, cc, dd
> > "aa/bb/cc/d1" was tokenized into 3 terms: aa, bb, cc/d1
> > "aa/bb/c1/dd" was tokenized into 2 terms: aa, bb/c1/dd
> > "aa/b1/cc/dd" was tokenized into 2 terms: aa/b1/cc, dd
> > "a1/bb/cc/dd" was tokenized into 3 terms: a1/bb, cc, dd
> >
> > It seems that if the input string includes a numerical value, any first
preceeding and/or next following slash ('/') is treated as a character.
Otherwise the slash is apparently treated as a token separator.
> >
> > I'm lost. Assuming this is not a bug, could somebody explain the rhyme
and reason to this tokenizing logic?
>
> This is a heuristic that tries to index alphanumeric model and serial
> numbers as a single token, but not to index long hyphenated or slashed
> phrases as a single token. It requires digits in at least every other
> slash or dash-delimted segment.
>
> Perhaps it is misguided. Can you suggest a better heuristic?
>
> The challenge is to index things like "B-17", "F/A-18", "PS/2",
> "802.11a", "0-85152-629-2", etc. as single tokens, but not to index
> things like "once-famous-but-now-forgotten" or
> "red/orange/yellow/green/blue/indigo/violet" as single tokens.
>
> Doug
>
>
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Incomprehensible (to me) tokenizing behavior
Posted by Doug Cutting <cu...@lucene.com>.
Terry Steichen wrote:
> I tested StandardAnalyzer (which uses StandardTokenizer) by inputing the a set of strings which produced the following results:
>
> "aa/bb/cc/dd" was tokenized into 4 terms: aa, bb, cc, dd
> "aa/bb/cc/d1" was tokenized into 3 terms: aa, bb, cc/d1
> "aa/bb/c1/dd" was tokenized into 2 terms: aa, bb/c1/dd
> "aa/b1/cc/dd" was tokenized into 2 terms: aa/b1/cc, dd
> "a1/bb/cc/dd" was tokenized into 3 terms: a1/bb, cc, dd
>
> It seems that if the input string includes a numerical value, any first preceeding and/or next following slash ('/') is treated as a character. Otherwise the slash is apparently treated as a token separator.
>
> I'm lost. Assuming this is not a bug, could somebody explain the rhyme and reason to this tokenizing logic?
This is a heuristic that tries to index alphanumeric model and serial
numbers as a single token, but not to index long hyphenated or slashed
phrases as a single token. It requires digits in at least every other
slash or dash-delimted segment.
Perhaps it is misguided. Can you suggest a better heuristic?
The challenge is to index things like "B-17", "F/A-18", "PS/2",
"802.11a", "0-85152-629-2", etc. as single tokens, but not to index
things like "once-famous-but-now-forgotten" or
"red/orange/yellow/green/blue/indigo/violet" as single tokens.
Doug
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>