You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Steven Rowe (JIRA)" <ji...@apache.org> on 2010/01/17 18:17:54 UTC
[jira] Issue Comment Edited: (LUCENE-2218) ShingleFilter
improvements
[ https://issues.apache.org/jira/browse/LUCENE-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801341#action_12801341 ]
Steven Rowe edited comment on LUCENE-2218 at 1/17/10 5:17 PM:
--------------------------------------------------------------
The rewrite included some optimizations (e.g., no longer constructing n StringBuilders for every position in the input stream), and the performance is now modestly better - below is a comparison generated using the attached benchmark patch:
JAVA:
java version "1.5.0_15"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_15-b04, mixed mode)
OS:
cygwin
WinVistaService Pack 2
Service Pack 26060022202561
||Max Shingle Size||Unigrams?||Unpatched||Patched||StandardAnalyzer||Improvement||
|2|no|4.92s|4.74s|2.19s|7.5%|
|2|yes|5.04s|4.90s|2.19s|5.6%|
|4|no|6.21s|5.82s|2.19s|11.2%|
|4|yes|6.41s|5.97s|2.19s|12.1%|
was (Author: steve_rowe):
The rewrite included some optimizations (e.g., no longer constructing n StringBuilders for every position in the input stream), and the performance is now modestly better - below is a comparison generated using the attached benchmark patch:
JAVA:
java version "1.5.0_15"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_15-b04, mixed mode)
OS:
cygwin
WinVistaService Pack 2
Service Pack 26060022202561
||Max Shingle Size||Unigrams?||Unpatched||Patched||StandardAnalyzer||Improvement||
|2|no|4.92|4.74|2.19|7.5%|
|2|yes|5.04|4.90|2.19|5.6%|
|4|no|6.21|5.82|2.19|11.2%|
|4|yes|6.41|5.97|2.19|12.1%|
> ShingleFilter improvements
> --------------------------
>
> Key: LUCENE-2218
> URL: https://issues.apache.org/jira/browse/LUCENE-2218
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 3.0
> Reporter: Steven Rowe
> Priority: Minor
> Attachments: LUCENE-2218.benchmark.patch, LUCENE-2218.benchmark.patch, LUCENE-2218.patch
>
>
> ShingleFilter should allow configuration of minimum shingle size (in addition to maximum shingle size), so that it's possible to (e.g.) output only trigrams instead of bigrams mixed with trigrams. The token separator used in composing shingles should be configurable too.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org