You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Mike Snare <mi...@gmail.com> on 2004/12/15 19:29:51 UTC
Why does the StandardTokenizer split hyphenated words?
I am writing a tool that uses lucene, and I immediately ran into a
problem searching for words that contain internal hyphens (dashes).
After looking at the StandardTokenizer, I saw that it was because
there is no rule that will match <ALPHA> <P> <ALPHA> or <ALPHANUM> <P>
<ALPHANUM>. Based on what I can tell from the source, every other
term in a word containing any of the following (.,/-_) must contain at
least one digit.
I was wondering if someone could shed some light on why it was deemed
necessary to prevent indexing a word like 'word-with-hyphen' without
first splitting it into its constituent parts. The only reason I can
think of (and the only one I've found) is to handle hyphenated words
at line breaks, although my first thought would be that this would be
undesired behavior, since a word that was broken due to a line break
should actually be reconstructed, and not split.
In my case, the words are keywords that must remain as is, searchable
with the hyphen in place. It was easy enough to modify the tokenizer
to do what I need, so I'm not really asking for help there. I'm
really just curious as to why it is that "a-1" is considered a single
token, but "a-b" is split.
Anyone care to elaborate?
Thanks,
-Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Why does the StandardTokenizer split hyphenated words?
Posted by Chris Hostetter <ho...@fucit.org>.
: a-1 is considered a typical product name that needs to be unchanged
: (there's a comment in the source that mentions this). Indexing
: "hyphen-word" as two tokens has the advantage that it can then be found
: with the following queries:
: hypen-word (will be turned into a phrase query internally)
: "hypen word" (phrase query)
: (it cannot be found searching for hyphenword, however).
This isn't an area of Lucene that I've had a chance to investigate much
yet, but if I recall from my reading, Lucene allows you to place multiple
token sequences at the same position, generating something more easily
described as a "token graph" then a "token stream" .. correct?
so given an input "the quick-brown fox jumped over the a-1 sauce"
the tokenizer culd generate a token stream that looks like.
"the"
"quick" "brown" OR "quick-brown" OR "quickbrown"
"fox"
"jumped"
"over"
"the"
"a" "1" OR "a-1" OR "a1"
"sauce"
...at which point, a minimum 2 character word length filter, and stop
words filter could (if you wanted to use them) reduce that to...
"quick" "brown" OR "quick-brown" OR "quickbrown"
"fox"
"jumped"
"over"
"a-1" OR "a1"
"sauce"
allowing all of these future (phrase) searches to match...
the quick brown fox jumped over the a1 sauce
the quickbrown fox jumped over the a1 sauce
the quick-brown fox jumped over the a1 sauce
the quick brown fox jumped over the a 1 sauce
the quickbrown fox jumped over the a 1 sauce
the quick-brown fox jumped over the a 1 sauce
the quick brown fox jumped over the a-1 sauce
the quickbrown fox jumped over the a-1 sauce
the quick-brown fox jumped over the a-1 sauce
...correct? or am I missunderstanding?
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Why does the StandardTokenizer split hyphenated words?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Dec 15, 2004, at 3:14 PM, Mike Snare wrote:
[...]
>
> In addition, why do we assume that a-1 is a "typical product name" but
> a-b isn't?
>
> I am in no way second-guessing or suggesting a change, It just doesn't
> make sense to me, and I'm trying to understand. It is very likely, as
> is oft the case, that this is just one of those things one has to
> accept.
>
It is one of those things we have to accept... or in this case write
our own analyzer. An Analyzer is a very special and custom choice.
StandardAnalyzer is a general purpose one, but quite insufficient in
many cases. Like QueryParser. We're lucky to have these kitchen-sink
pieces in Lucene to get us going quickly, but digging deeper we often
need custom solutions.
I'm working on indexing the e-book of Lucene in Action. I'll blog up
the details of this in the near future as case-study material, but
here's the short version...
I got the PDF file, ran pdftotext on it. Many words are split across
lines with a hyphen. Often these pieces should be combined with the
hyphen removed. Sometimes, though, these words are to be split. The
scenario is different than yours, because I want the hyphens gone -
though sometimes they are a separator and sometimes they should be
removed. It depends. I wrote a custom analyzer with several custom
filters in the pipeline... dashes are originally kept in the stream,
and a later filter combines two tokens and looks it up in an exception
list and either combines it or leaves it separate. StandardAnalyzer
would have wreaked havoc.
The results of my work will soon be available to all to poke at, but
for now a screenshot is all I have public:
http://www.lucenebook.com
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Why does the StandardTokenizer split hyphenated words?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hello,
As Erik already said - that Analyzer is really there to get people
going quickly and as a 'does pretty good' Analyzer. There is no
Analyzer that will work for everyone, and Analyzers are meant to be
custom-made. It looks like you already got that figured out and have
your own Analyzer.
Otis
--- Mike Snare <mi...@gmail.com> wrote:
> Absolutely, but -- correct me if I'm wrong -- it would give no higher
> ranking to half-baked and would take a good deal longer on large
> indices.
>
>
> On Thu, 16 Dec 2004 20:03:27 +0100, Daniel Naber
> <da...@t-online.de> wrote:
> > On Thursday 16 December 2004 13:46, Mike Snare wrote:
> >
> > > > Maybe for "a-b", but what about English words like
> "half-baked"?
> > >
> > > Perhaps that's the difference in thinking, then. I would imagine
> that
> > > you would want to search on "half-baked" and not "half AND
> baked".
> >
> > A search for half-baked will find both half-baked and "half baked"
> (the
> > phrase). The only thing you'll not find if halfbaked.
> >
> > Regards
> > Daniel
> >
> > --
> > http://www.danielnaber.de
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Why does the StandardTokenizer split hyphenated words?
Posted by Mike Snare <mi...@gmail.com>.
Absolutely, but -- correct me if I'm wrong -- it would give no higher
ranking to half-baked and would take a good deal longer on large
indices.
On Thu, 16 Dec 2004 20:03:27 +0100, Daniel Naber
<da...@t-online.de> wrote:
> On Thursday 16 December 2004 13:46, Mike Snare wrote:
>
> > > Maybe for "a-b", but what about English words like "half-baked"?
> >
> > Perhaps that's the difference in thinking, then. I would imagine that
> > you would want to search on "half-baked" and not "half AND baked".
>
> A search for half-baked will find both half-baked and "half baked" (the
> phrase). The only thing you'll not find if halfbaked.
>
> Regards
> Daniel
>
> --
> http://www.danielnaber.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Why does the StandardTokenizer split hyphenated words?
Posted by Daniel Naber <da...@t-online.de>.
On Thursday 16 December 2004 13:46, Mike Snare wrote:
> > Maybe for "a-b", but what about English words like "half-baked"?
>
> Perhaps that's the difference in thinking, then. I would imagine that
> you would want to search on "half-baked" and not "half AND baked".
A search for half-baked will find both half-baked and "half baked" (the
phrase). The only thing you'll not find if halfbaked.
Regards
Daniel
--
http://www.danielnaber.de
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Why does the StandardTokenizer split hyphenated words?
Posted by Mike Snare <mi...@gmail.com>.
> Not if these words are spelling variations of the same concept, which
> doesn't seem unlikely.
>
> > In addition, why do we assume that a-1 is a "typical product name" but
> > a-b isn't?
>
> Maybe for "a-b", but what about English words like "half-baked"?
Perhaps that's the difference in thinking, then. I would imagine that
you would want to search on "half-baked" and not "half AND baked".
> Regards
> Daniel
>
> --
> http://www.danielnaber.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Why does the StandardTokenizer split hyphenated words?
Posted by Daniel Naber <da...@t-online.de>.
On Wednesday 15 December 2004 21:14, Mike Snare wrote:
> Also, the phrase query
> would place the same value on a doc that simply had the two words as a
> doc that had the hyphenated version, wouldn't it? This seems odd.
Not if these words are spelling variations of the same concept, which
doesn't seem unlikely.
> In addition, why do we assume that a-1 is a "typical product name" but
> a-b isn't?
Maybe for "a-b", but what about English words like "half-baked"?
Regards
Daniel
--
http://www.danielnaber.de
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Why does the StandardTokenizer split hyphenated words?
Posted by Mike Snare <mi...@gmail.com>.
> a-1 is considered a typical product name that needs to be unchanged
> (there's a comment in the source that mentions this). Indexing
> "hyphen-word" as two tokens has the advantage that it can then be found
> with the following queries:
> hypen-word (will be turned into a phrase query internally)
> "hypen word" (phrase query)
> (it cannot be found searching for hyphenword, however).
Sure. But phrase queries are slower than a single word query. In my
case, using the standard analyzer prior to my modification caused a
single (hyphenated) word query to take upwards of 10 seconds (1M+
documents with ~400K terms). The exact same search with the new
Analyzer takes <.5 seconds (granted the new tokenization caused a
significant reduction in the number of terms). Also, the phrase query
would place the same value on a doc that simply had the two words as a
doc that had the hyphenated version, wouldn't it? This seems odd.
In addition, why do we assume that a-1 is a "typical product name" but
a-b isn't?
I am in no way second-guessing or suggesting a change, It just doesn't
make sense to me, and I'm trying to understand. It is very likely, as
is oft the case, that this is just one of those things one has to
accept.
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Why does the StandardTokenizer split hyphenated words?
Posted by Daniel Naber <da...@t-online.de>.
On Wednesday 15 December 2004 19:29, Mike Snare wrote:
> In my case, the words are keywords that must remain as is, searchable
> with the hyphen in place. It was easy enough to modify the tokenizer
> to do what I need, so I'm not really asking for help there. I'm
> really just curious as to why it is that "a-1" is considered a single
> token, but "a-b" is split.
a-1 is considered a typical product name that needs to be unchanged
(there's a comment in the source that mentions this). Indexing
"hyphen-word" as two tokens has the advantage that it can then be found
with the following queries:
hypen-word (will be turned into a phrase query internally)
"hypen word" (phrase query)
(it cannot be found searching for hyphenword, however).
Regards
Daniel
--
http://www.danielnaber.de
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org