You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Martin O'Shea <ap...@dsl.pipex.com> on 2010/10/24 21:58:41 UTC
Use of hyphens in StandardAnalyzer
Hello
I have a StandardAnalyzer working which retrieves words and frequencies from
a single document using a TermVectorMapper which is populating a HashMap.
But if I use the following text as a field in my document, i.e.
addDoc(w, "lucene Lawton-Browne Lucene");
The word frequencies returned in the HashMap are:
browne 1
lucene 2
lawton 1
The problem is the words 'lawton' and 'browne'. If this is an actual
'double-barreled' name, can Lucene recognise it as 'Lawton-Browne' where the
name is actually a single word?
I've tried combinations of:
addDoc(w, "lucene \"Lawton-Browne\" Lucene");
And single quotes but without success.
Thanks
Martin O'Shea.
RE: Use of hyphens in StandardAnalyzer
Posted by Martin O'Shea <ap...@dsl.pipex.com>.
A good suggestion. But I'm using Lucene 3.2 and the constructor for a StandardAnalyzer has Version_30 as its highest value.
-----Original Message-----
From: Steven A Rowe [mailto:sarowe@syr.edu]
Sent: 24 Oct 2010 21 31
To: java-user@lucene.apache.org
Subject: RE: Use of hyphens in StandardAnalyzer
Hi Martin,
StandardTokenizer and -Analyzer have been changed, as of future version 3.1 (the next release) to support the Unicode segmentation rules in UAX#29. My (untested) guess is that your hyphenated word will be kept as a single token if you set the version to 3.1 or higher in the constructor.
Steve
> -----Original Message-----
> From: Martin O'Shea [mailto:appy74@dsl.pipex.com]
> Sent: Sunday, October 24, 2010 3:59 PM
> To: java-user@lucene.apache.org
> Subject: Use of hyphens in StandardAnalyzer
>
> Hello
>
>
>
> I have a StandardAnalyzer working which retrieves words and frequencies
> from
> a single document using a TermVectorMapper which is populating a HashMap.
>
>
>
> But if I use the following text as a field in my document, i.e.
>
>
>
> addDoc(w, "lucene Lawton-Browne Lucene");
>
>
>
> The word frequencies returned in the HashMap are:
>
>
>
> browne 1
>
> lucene 2
>
> lawton 1
>
>
>
> The problem is the words 'lawton' and 'browne'. If this is an actual
> 'double-barreled' name, can Lucene recognise it as 'Lawton-Browne' where
> the
> name is actually a single word?
>
>
>
> I've tried combinations of:
>
>
>
> addDoc(w, "lucene \"Lawton-Browne\" Lucene");
>
>
>
> And single quotes but without success.
>
>
>
> Thanks
>
>
>
> Martin O'Shea.
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Use of hyphens in StandardAnalyzer
Posted by Steven A Rowe <sa...@syr.edu>.
Hi Martin,
StandardTokenizer and -Analyzer have been changed, as of future version 3.1 (the next release) to support the Unicode segmentation rules in UAX#29. My (untested) guess is that your hyphenated word will be kept as a single token if you set the version to 3.1 or higher in the constructor.
Steve
> -----Original Message-----
> From: Martin O'Shea [mailto:appy74@dsl.pipex.com]
> Sent: Sunday, October 24, 2010 3:59 PM
> To: java-user@lucene.apache.org
> Subject: Use of hyphens in StandardAnalyzer
>
> Hello
>
>
>
> I have a StandardAnalyzer working which retrieves words and frequencies
> from
> a single document using a TermVectorMapper which is populating a HashMap.
>
>
>
> But if I use the following text as a field in my document, i.e.
>
>
>
> addDoc(w, "lucene Lawton-Browne Lucene");
>
>
>
> The word frequencies returned in the HashMap are:
>
>
>
> browne 1
>
> lucene 2
>
> lawton 1
>
>
>
> The problem is the words 'lawton' and 'browne'. If this is an actual
> 'double-barreled' name, can Lucene recognise it as 'Lawton-Browne' where
> the
> name is actually a single word?
>
>
>
> I've tried combinations of:
>
>
>
> addDoc(w, "lucene \"Lawton-Browne\" Lucene");
>
>
>
> And single quotes but without success.
>
>
>
> Thanks
>
>
>
> Martin O'Shea.
>
>
>
>
>
>