You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Bernd Fehling (JIRA)" <ji...@apache.org> on 2011/07/01 09:40:28 UTC
[jira] [Created] (SOLR-2628) use of FST for SynonymsFilterFactory
and synonyms.txt
use of FST for SynonymsFilterFactory and synonyms.txt
-----------------------------------------------------
Key: SOLR-2628
URL: https://issues.apache.org/jira/browse/SOLR-2628
Project: Solr
Issue Type: New Feature
Components: Schema and Analysis
Affects Versions: 3.4, 4.0
Environment: Linux
Reporter: Bernd Fehling
Priority: Minor
Currently the SynonymsFilterFactory builds up a memory based SynonymsMap.
This can generate huge maps because of the permutations for synonyms.
Now where FST (finite state transducer) is introduced to lucene this could also be used for synonyms.
A tool can compile the synoynms.txt file to a binary automaton file which can then be used
with SynoynmsFilterFactory.
Advantage:
- faster start of solr, no need to generate SynonymsMap
- faster lookup
- memory saving
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [jira] [Created] (SOLR-2628) use of FST for SynonymsFilterFactory
and synonyms.txt
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
> I don't need information about solr projects.
I'm afraid you don't have much choice because Solr and Lucene coexist
in the same mailing list/ Jira space.
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [jira] [Created] (SOLR-2628) use of FST for SynonymsFilterFactory
and synonyms.txt
Posted by "yazhini.k vini" <ya...@gmail.com>.
I don't need information about solr projects.
*
*
**yazhini**
[jira] [Assigned] (SOLR-2628) use of FST for SynonymsFilterFactory
and synonyms.txt
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dawid Weiss reassigned SOLR-2628:
---------------------------------
Assignee: Dawid Weiss
> use of FST for SynonymsFilterFactory and synonyms.txt
> -----------------------------------------------------
>
> Key: SOLR-2628
> URL: https://issues.apache.org/jira/browse/SOLR-2628
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Affects Versions: 3.4, 4.0
> Environment: Linux
> Reporter: Bernd Fehling
> Assignee: Dawid Weiss
> Priority: Minor
> Labels: suggestion
>
> Currently the SynonymsFilterFactory builds up a memory based SynonymsMap.
> This can generate huge maps because of the permutations for synonyms.
> Now where FST (finite state transducer) is introduced to lucene this could also be used for synonyms.
> A tool can compile the synoynms.txt file to a binary automaton file which can then be used
> with SynoynmsFilterFactory.
> Advantage:
> - faster start of solr, no need to generate SynonymsMap
> - faster lookup
> - memory saving
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2628) use of FST for SynonymsFilterFactory
and synonyms.txt
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058470#comment-13058470 ]
Dawid Weiss commented on SOLR-2628:
-----------------------------------
Yep, this is a duplicate. Thanks Mike. Like I said -- I won't be able to work on this for the next two weeks (I also have that FST refactoring opened up in the background... it's progressing slowly), but it's definitely a low-hanging fruit to pick because it shouldn't be very difficult and the gains huge.
> use of FST for SynonymsFilterFactory and synonyms.txt
> -----------------------------------------------------
>
> Key: SOLR-2628
> URL: https://issues.apache.org/jira/browse/SOLR-2628
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Affects Versions: 3.4, 4.0
> Environment: Linux
> Reporter: Bernd Fehling
> Assignee: Dawid Weiss
> Priority: Minor
> Labels: suggestion
>
> Currently the SynonymsFilterFactory builds up a memory based SynonymsMap.
> This can generate huge maps because of the permutations for synonyms.
> Now where FST (finite state transducer) is introduced to lucene this could also be used for synonyms.
> A tool can compile the synoynms.txt file to a binary automaton file which can then be used
> with SynoynmsFilterFactory.
> Advantage:
> - faster start of solr, no need to generate SynonymsMap
> - faster lookup
> - memory saving
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2628) use of FST for SynonymsFilterFactory
and synonyms.txt
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058468#comment-13058468 ]
Michael McCandless commented on SOLR-2628:
------------------------------------------
Dawid, have a look at LUCENE-3233 -- we have a [very very rough] start at this.
> use of FST for SynonymsFilterFactory and synonyms.txt
> -----------------------------------------------------
>
> Key: SOLR-2628
> URL: https://issues.apache.org/jira/browse/SOLR-2628
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Affects Versions: 3.4, 4.0
> Environment: Linux
> Reporter: Bernd Fehling
> Assignee: Dawid Weiss
> Priority: Minor
> Labels: suggestion
>
> Currently the SynonymsFilterFactory builds up a memory based SynonymsMap.
> This can generate huge maps because of the permutations for synonyms.
> Now where FST (finite state transducer) is introduced to lucene this could also be used for synonyms.
> A tool can compile the synoynms.txt file to a binary automaton file which can then be used
> with SynoynmsFilterFactory.
> Advantage:
> - faster start of solr, no need to generate SynonymsMap
> - faster lookup
> - memory saving
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (SOLR-2628) use of FST for SynonymsFilterFactory
and synonyms.txt
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dawid Weiss resolved SOLR-2628.
-------------------------------
Resolution: Duplicate
Duplicate of LUCENE-3233
> use of FST for SynonymsFilterFactory and synonyms.txt
> -----------------------------------------------------
>
> Key: SOLR-2628
> URL: https://issues.apache.org/jira/browse/SOLR-2628
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Affects Versions: 3.4, 4.0
> Environment: Linux
> Reporter: Bernd Fehling
> Assignee: Dawid Weiss
> Priority: Minor
> Labels: suggestion
>
> Currently the SynonymsFilterFactory builds up a memory based SynonymsMap.
> This can generate huge maps because of the permutations for synonyms.
> Now where FST (finite state transducer) is introduced to lucene this could also be used for synonyms.
> A tool can compile the synoynms.txt file to a binary automaton file which can then be used
> with SynoynmsFilterFactory.
> Advantage:
> - faster start of solr, no need to generate SynonymsMap
> - faster lookup
> - memory saving
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2628) use of FST for SynonymsFilterFactory
and synonyms.txt
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058241#comment-13058241 ]
Dawid Weiss commented on SOLR-2628:
-----------------------------------
I've talked about it a little bit with Bernd and indeed, it seems possible to reduce the size of in-memory data structures by an order of magnitude (or even two orders of magnitude, we shall see). I'm on vacation for the next week and on a business trip for another one after that, but I'll be on it once I come back home.
> use of FST for SynonymsFilterFactory and synonyms.txt
> -----------------------------------------------------
>
> Key: SOLR-2628
> URL: https://issues.apache.org/jira/browse/SOLR-2628
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Affects Versions: 3.4, 4.0
> Environment: Linux
> Reporter: Bernd Fehling
> Assignee: Dawid Weiss
> Priority: Minor
> Labels: suggestion
>
> Currently the SynonymsFilterFactory builds up a memory based SynonymsMap.
> This can generate huge maps because of the permutations for synonyms.
> Now where FST (finite state transducer) is introduced to lucene this could also be used for synonyms.
> A tool can compile the synoynms.txt file to a binary automaton file which can then be used
> with SynoynmsFilterFactory.
> Advantage:
> - faster start of solr, no need to generate SynonymsMap
> - faster lookup
> - memory saving
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2628) use of FST for SynonymsFilterFactory
and synonyms.txt
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058475#comment-13058475 ]
Michael McCandless commented on SOLR-2628:
------------------------------------------
I think the reduction of RAM should be huge but lookup speed might be slower (ie the usual tradeoff of FST), since we are going char by char in the FST. If we go word-by-word (ie FST's labels are word ords and we separately resolve word -> ord via "normal" hash lookup) then that might be a good middle ground... but this is all speculation for now!
> use of FST for SynonymsFilterFactory and synonyms.txt
> -----------------------------------------------------
>
> Key: SOLR-2628
> URL: https://issues.apache.org/jira/browse/SOLR-2628
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Affects Versions: 3.4, 4.0
> Environment: Linux
> Reporter: Bernd Fehling
> Assignee: Dawid Weiss
> Priority: Minor
> Labels: suggestion
>
> Currently the SynonymsFilterFactory builds up a memory based SynonymsMap.
> This can generate huge maps because of the permutations for synonyms.
> Now where FST (finite state transducer) is introduced to lucene this could also be used for synonyms.
> A tool can compile the synoynms.txt file to a binary automaton file which can then be used
> with SynoynmsFilterFactory.
> Advantage:
> - faster start of solr, no need to generate SynonymsMap
> - faster lookup
> - memory saving
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2628) use of FST for SynonymsFilterFactory
and synonyms.txt
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058489#comment-13058489 ]
Dawid Weiss commented on SOLR-2628:
-----------------------------------
Yes, this may be the case. It'd need to be investigated because storing words in a hashtable will also bump memory requirements, whereas an FST can at least reuse some prefixes and suffixes.
> use of FST for SynonymsFilterFactory and synonyms.txt
> -----------------------------------------------------
>
> Key: SOLR-2628
> URL: https://issues.apache.org/jira/browse/SOLR-2628
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Affects Versions: 3.4, 4.0
> Environment: Linux
> Reporter: Bernd Fehling
> Assignee: Dawid Weiss
> Priority: Minor
> Labels: suggestion
>
> Currently the SynonymsFilterFactory builds up a memory based SynonymsMap.
> This can generate huge maps because of the permutations for synonyms.
> Now where FST (finite state transducer) is introduced to lucene this could also be used for synonyms.
> A tool can compile the synoynms.txt file to a binary automaton file which can then be used
> with SynoynmsFilterFactory.
> Advantage:
> - faster start of solr, no need to generate SynonymsMap
> - faster lookup
> - memory saving
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org