You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by ash nix <ni...@gmail.com> on 2013/02/28 19:04:24 UTC
lucene 4 index
Hi,
Can anyone please send me document on lucene 4 index format?
Want to know internals of index.
--
Thanks,
A
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Migrating SnowballAnalyzer to 4.1
Posted by Robert Muir <rc...@gmail.com>.
On Sat, Mar 16, 2013 at 12:57 AM, Steve Rowe <sa...@gmail.com> wrote:
>
> Thanks for the explanation.
>
> I ran a lucene/benchmark alg comparing the two stemmers on trunk on my Macbook Pro with Oracle Java 1.7.0_13, and it looks like the situation hasn't changed much.
>
> The original-algorithm Porter stemmer is 4 times faster than the Porter2/English Snowball stemmer, resulting in 40% higher throughput in a full English analysis pipeline.
>
> So the default English stemmer choice is still valid IMO.
>
Thanks a lot for running this!
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Migrating SnowballAnalyzer to 4.1
Posted by Steve Rowe <sa...@gmail.com>.
Hi Robert,
On Mar 15, 2013, at 11:29 AM, Robert Muir <rc...@gmail.com> wrote:
> 2013/2/28 Steve Rowe <sa...@gmail.com>:
>> EnglishAnalyzer has used PorterStemmer instead of the English Snowball stemmer since it was created in 2010 as part of LUCENE-2055[2]. I think this is an oversight: EnglishAnalyzer should incorporate the best English stemmer we've got, and Martin Porter says the Porter2 stemmer is better[1]. Robert Muir (who wrote EnglishAnalyzer), if you're reading, what do you think?
>
> This was intentional actually. The default was a tradeoff of
> "benefits" (which affect less than 5% of english vocabulary, if you
> read around the snowball site), versus a much more significant
> performance difference as a "default".
>
> For example when i did tests of indexing both short and long texts
>
> http://find.searchhub.org/document/c1d3301b71dab5ca#46a8351089a98aec
>
> Thats overall indexing speed, not just text analysis.
>
> It might be that this guy is faster these days (we've done some
> improvements) too.
Thanks for the explanation.
I ran a lucene/benchmark alg comparing the two stemmers on trunk on my Macbook Pro with Oracle Java 1.7.0_13, and it looks like the situation hasn't changed much.
The original-algorithm Porter stemmer is 4 times faster than the Porter2/English Snowball stemmer, resulting in 40% higher throughput in a full English analysis pipeline.
So the default English stemmer choice is still valid IMO.
Here's porter-comparison.alg:
-----
content.source=org.apache.lucene.benchmark.byTask.feeds.ReutersContentSource
doc.tokenized=false
doc.body.tokenized=true
docs.dir=reuters-out
-AnalyzerFactory(name:original-porter-stemmer,StandardTokenizer,
StandardFilter,EnglishPossessiveFilter,LowerCaseFilter,StopFilter,
PorterStemFilter)
-AnalyzerFactory(name:porter2-stemmer,StandardTokenizer,
StandardFilter,EnglishPossessiveFilter,LowerCaseFilter,StopFilter,
SnowballPorterFilter(language:English))
-AnalyzerFactory(name:no-stemmer,StandardTokenizer,
StandardFilter,EnglishPossessiveFilter,LowerCaseFilter,StopFilter)
{ "Rounds"
-NewAnalyzer(original-porter-stemmer)
-ResetInputs
{ "Original Porter Stemmer" { ReadTokens > : 20000 }
-NewAnalyzer(porter2-stemmer)
-ResetInputs
{ "Porter2/English Stemmer" { ReadTokens > : 20000 }
-NewAnalyzer(no-stemmer)
-ResetInputs
{ "No Stemmer" { ReadTokens > : 20000 }
NewRound
} : 5
RepSumByNameRound
-----
And the results (regrouped; ordered by elapsedSec) - a "rec" is a token:
-----
Operation round recsPerRun rec/s elapsedSec
No Stemmer 2 1814029 1,234,873.38 1.47
No Stemmer 4 1814029 1,234,873.38 1.47
No Stemmer 1 1814029 1,230,684.50 1.47
No Stemmer 0 1814029 1,227,353.88 1.48
No Stemmer 3 1814029 1,226,524.00 1.48
Original Porter Stemmer 1 1814029 1,074,025.50 1.69
Original Porter Stemmer 4 1814029 1,065,196.12 1.70
Original Porter Stemmer 2 1814029 1,056,510.75 1.72
Original Porter Stemmer 3 1814029 1,030,698.31 1.76
Original Porter Stemmer 0 1814029 685,833.25 2.64
Porter2/English Stemmer 4 1814029 768,656.38 2.36
Porter2/English Stemmer 2 1814029 764,123.44 2.37
Porter2/English Stemmer 1 1814029 758,056.44 2.39
Porter2/English Stemmer 3 1814029 758,056.44 2.39
Porter2/English Stemmer 0 1814029 716,158.31 2.53
-----
Best of 5 results:
No Stemmer: 1.47s
Original Porter Stemmer: 1.69s - 1.47s = 0.22s
Porter2/English Stemmer: 2.36s - 1.47s = 0.89s
Throughput increase: (2.36s-1.69s)/1.69s * 100 = 40%
Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Migrating SnowballAnalyzer to 4.1
Posted by Robert Muir <rc...@gmail.com>.
2013/2/28 Steve Rowe <sa...@gmail.com>:
> EnglishAnalyzer has used PorterStemmer instead of the English Snowball stemmer since it was created in 2010 as part of LUCENE-2055[2]. I think this is an oversight: EnglishAnalyzer should incorporate the best English stemmer we've got, and Martin Porter says the Porter2 stemmer is better[1]. Robert Muir (who wrote EnglishAnalyzer), if you're reading, what do you think?
This was intentional actually. The default was a tradeoff of
"benefits" (which affect less than 5% of english vocabulary, if you
read around the snowball site), versus a much more significant
performance difference as a "default".
For example when i did tests of indexing both short and long texts
http://find.searchhub.org/document/c1d3301b71dab5ca#46a8351089a98aec
Thats overall indexing speed, not just text analysis.
It might be that this guy is faster these days (we've done some
improvements) too.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Migrating SnowballAnalyzer to 4.1
Posted by Steve Rowe <sa...@gmail.com>.
Hi Peng,
The short answer: EnglishAnalyzer will behave differently in terms of stemming than SnowballAnalyzer("English", StopAnalyzer.ENGLISH_STOP_WORDS).
PorterStemmer, which is used by the EnglishAnalyzer in analyzers-common, is an older version of the English Snowball stemmer (now called EnglishStemmer in Lucene). See the Status section in Martin Porter's page[1] for a description of the differences - PorterStemmer implements his original algorithm, and EnglishStemmer implements the Porter2 algorithm.
EnglishAnalyzer has used PorterStemmer instead of the English Snowball stemmer since it was created in 2010 as part of LUCENE-2055[2]. I think this is an oversight: EnglishAnalyzer should incorporate the best English stemmer we've got, and Martin Porter says the Porter2 stemmer is better[1]. Robert Muir (who wrote EnglishAnalyzer), if you're reading, what do you think?
If you want a drop in replacement, for now it looks like you'll have to put it together yourself. It's not hard - you can copy the source for EnglishAnalyzer[1] and substitute 'SnowballFilter(result, "English")' where you see 'PorterStemFilter(result)', and import org.apache.lucene.analysis.snowball.SnowballFilter.
Steve
[1] Martin Porter's Porter Stemming Algorithm page: <http://tartarus.org/martin/PorterStemmer/index.html>
[2] LUCENE-2055: <https://issues.apache.org/jira/browse/LUCENE-2055>
[3] EnglishAnalyzer source code: <http://svn.apache.org/repos/asf/lucene/dev/tags/lucene_solr_4_1_0/lucene/analysis/common/src/java/org/apache/lucene/analysis/en/EnglishAnalyzer.java>
As far as I can tell,
On Feb 28, 2013, at 2:52 PM, Peng Gao <pg...@esri.com> wrote:
> Hi Steve,
> Thanks for the help. One more question:
> Is EnglishAnalyzer a drop-in replacement for SnowballAnalyzer("English", ...), in terms
> of stemming?
>
>
> Thanks again
> Peng
>
> PS
> Sorry for the Thread Hijacking. Will behave the next time.
>
>> -----Original Message-----
>> From: Steve Rowe [mailto:sarowe@gmail.com]
>> Sent: Thursday, February 28, 2013 10:47 AM
>> To: java-user@lucene.apache.org
>> Subject: Re: Migrating SnowballAnalyzer to 4.1
>>
>> Hi Peng,
>>
>> Take a look at the release docs:
>> <http://lucene.apache.org/core/4_1_0/index.html>
>>
>> In particular, in the API Javadocs section, the analyzers-common
>> documentation has a large list of per-language analyzers. EnglishAnalyzer
>> is under the org.apache.lucene.analysis.en package:
>> <http://lucene.apache.org/core/4_1_0/analyzers-
>> common/org/apache/lucene/analysis/en/package-summary.html>
>>
>> Steve
>>
>> On Feb 28, 2013, at 1:28 PM, Peng Gao <pg...@esri.com> wrote:
>>
>>> Hi,
>>>
>>> I have a Lucene 2.9.x app that uses
>>> org.apache.lucene.analysis.snowball.SnowballAnalyzer for index
>>> generation,
>>>
>>> analyzer = new SnowballAnalyzer("English",
>>> StopAnalyzer.ENGLISH_STOP_WORDS);
>>>
>>> and I want to upgrade it to 4.1.
>>>
>>> SnowballAnalyzer is deprecated in 4.1. The doc simply states
>>>
>>> "Deprecated. (3.1) Use the language-specific analyzer in
>>> modules/analysis instead. This analyzer will be removed in Lucene 5.0."
>>>
>>> I can't figure out how to rewrite it using 4.1 API. Could you help?
>>>
>>> Thanks,
>>> Peng
>>>
>>> Т
>>> ХF
>>> V 7V'67& &R R â f W6W" V 7V'67& &T V6V R 6 R &pФf "FF F
>>> 6 G2 R â f W6W"ֆV V6V R 6 R &pР
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Migrating SnowballAnalyzer to 4.1
Posted by Peng Gao <pg...@esri.com>.
Hi Steve,
Thanks for the help. One more question:
Is EnglishAnalyzer a drop-in replacement for SnowballAnalyzer("English", ...), in terms
of stemming?
Thanks again
Peng
PS
Sorry for the Thread Hijacking. Will behave the next time.
> -----Original Message-----
> From: Steve Rowe [mailto:sarowe@gmail.com]
> Sent: Thursday, February 28, 2013 10:47 AM
> To: java-user@lucene.apache.org
> Subject: Re: Migrating SnowballAnalyzer to 4.1
>
> Hi Peng,
>
> Take a look at the release docs:
> <http://lucene.apache.org/core/4_1_0/index.html>
>
> In particular, in the API Javadocs section, the analyzers-common
> documentation has a large list of per-language analyzers. EnglishAnalyzer
> is under the org.apache.lucene.analysis.en package:
> <http://lucene.apache.org/core/4_1_0/analyzers-
> common/org/apache/lucene/analysis/en/package-summary.html>
>
> Steve
>
> On Feb 28, 2013, at 1:28 PM, Peng Gao <pg...@esri.com> wrote:
>
> > Hi,
> >
> > I have a Lucene 2.9.x app that uses
> > org.apache.lucene.analysis.snowball.SnowballAnalyzer for index
> > generation,
> >
> > analyzer = new SnowballAnalyzer("English",
> > StopAnalyzer.ENGLISH_STOP_WORDS);
> >
> > and I want to upgrade it to 4.1.
> >
> > SnowballAnalyzer is deprecated in 4.1. The doc simply states
> >
> > "Deprecated. (3.1) Use the language-specific analyzer in
> > modules/analysis instead. This analyzer will be removed in Lucene 5.0."
> >
> > I can't figure out how to rewrite it using 4.1 API. Could you help?
> >
> > Thanks,
> > Peng
> >
> > Т
> > ХF
> > V 7V'67& &R R â f W6W" V 7V'67& &T V6V R 6 R &pФf "FF F
> > 6 G2 R â f W6W"ֆV V6V R 6 R &pР
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Migrating SnowballAnalyzer to 4.1
Posted by Steve Rowe <sa...@gmail.com>.
Hi Peng,
Take a look at the release docs: <http://lucene.apache.org/core/4_1_0/index.html>
In particular, in the API Javadocs section, the analyzers-common documentation has a large list of per-language analyzers. EnglishAnalyzer is under the org.apache.lucene.analysis.en package: <http://lucene.apache.org/core/4_1_0/analyzers-common/org/apache/lucene/analysis/en/package-summary.html>
Steve
On Feb 28, 2013, at 1:28 PM, Peng Gao <pg...@esri.com> wrote:
> Hi,
>
> I have a Lucene 2.9.x app that uses org.apache.lucene.analysis.snowball.SnowballAnalyzer for
> index generation,
>
> analyzer = new SnowballAnalyzer("English", StopAnalyzer.ENGLISH_STOP_WORDS);
>
> and I want to upgrade it to 4.1.
>
> SnowballAnalyzer is deprecated in 4.1. The doc simply states
>
> "Deprecated. (3.1) Use the language-specific analyzer in modules/analysis instead. This analyzer will be
> removed in Lucene 5.0."
>
> I can't figure out how to rewrite it using 4.1 API. Could you help?
>
> Thanks,
> Peng
>
> Т���������������������������������������������������������������������ХF�V�7V'67&�&R�R���â�f�W6W"�V�7V'67&�&T�V6V�R�6�R��&pФf�"FF�F����6����G2�R���â�f�W6W"ֆV��V6V�R�6�R��&pР�
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Migrating SnowballAnalyzer to 4.1
Posted by Chris Hostetter <ho...@fucit.org>.
: Subject: Migrating SnowballAnalyzer to 4.1
: References:
: <CA...@mail.gmail.com>
: <CA...@mail.gmail.com>
: In-Reply-To:
: <CA...@mail.gmail.com>
https://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Migrating SnowballAnalyzer to 4.1
Posted by Peng Gao <pg...@esri.com>.
Hi,
I have a Lucene 2.9.x app that uses org.apache.lucene.analysis.snowball.SnowballAnalyzer for
index generation,
analyzer = new SnowballAnalyzer("English", StopAnalyzer.ENGLISH_STOP_WORDS);
and I want to upgrade it to 4.1.
SnowballAnalyzer is deprecated in 4.1. The doc simply states
"Deprecated. (3.1) Use the language-specific analyzer in modules/analysis instead. This analyzer will be
removed in Lucene 5.0."
I can't figure out how to rewrite it using 4.1 API. Could you help?
Thanks,
Peng
Re: lucene 4 index
Posted by "Devon H. O'Dell" <de...@gmail.com>.
2013/2/28 ash nix <ni...@gmail.com>:
> Hi,
>
> Can anyone please send me document on lucene 4 index format?
> Want to know internals of index.
It is part of the Lucene documentation.
http://lucene.apache.org/core/4_1_0/core/org/apache/lucene/codecs/lucene41/package-summary.html#package_description
--dho
> --
> Thanks,
> A
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org