You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Martin Grotzke (JIRA)" <ji...@apache.org> on 2011/06/14 18:25:49 UTC
[jira] [Issue Comment Edited] (SOLR-2583) Make external scoring
more efficient (ExternalFileField, FileFloatSource)
[ https://issues.apache.org/jira/browse/SOLR-2583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049256#comment-13049256 ]
Martin Grotzke edited comment on SOLR-2583 at 6/14/11 4:25 PM:
---------------------------------------------------------------
I just compared memory consumption of the 3 different approaches, with different number of puts (number of scores) and sizes (number of docs), the memory is in byte:
{noformat}
Puts 1.000, size 1.000.000: CompactFloatArray 898.136, float[] 4.000.016, HashMap 72.192
Puts 10.000, size 1.000.000: CompactFloatArray 3.724.376, float[] 4.000.016, HashMap 702.784
Puts 100.000, size 1.000.000: CompactFloatArray 4.016.472, float[] 4.000.016, HashMap 6.607.808
Puts 1.000.000, size 1.000.000: CompactFloatArray 4.016.472, float[] 4.000.016, HashMap 44.644.032
Puts 1.000, size 5.000.000: CompactFloatArray 1.128.536, float[] 20.000.016, HashMap 72.256
Puts 10.000, size 5.000.000: CompactFloatArray 8.168.536, float[] 20.000.016, HashMap 704.832
Puts 100.000, size 5.000.000: CompactFloatArray 20.013.144, float[] 20.000.016, HashMap 7.385.152
Puts 1.000.000, size 5.000.000: CompactFloatArray 20.131.160, float[] 20.000.016, HashMap 66.395.584
Puts 1.000, size 10.000.000: CompactFloatArray 1.275.992, float[] 40.000.016, HashMap 72.256
Puts 10.000, size 10.000.000: CompactFloatArray 9.289.816, float[] 40.000.016, HashMap 705.280
Puts 100.000, size 10.000.000: CompactFloatArray 37.130.328, float[] 40.000.016, HashMap 7.418.112
Puts 1.000.000, size 10.000.000: CompactFloatArray 40.262.232, float[] 40.000.016, HashMap 69.282.496
{noformat}
I want to share this intermediately, without further interpretation/conclusion for now (I just need to get the train).
was (Author: martin.grotzke):
I just compared memory consumption of the 3 different approaches, with different number of puts (number of scores) and sizes (number of docs):
{noformat}
Puts 1.000, size 1.000.000: CompactFloatArray 898.136, float[] 4.000.016, HashMap 72.192
Puts 10.000, size 1.000.000: CompactFloatArray 3.724.376, float[] 4.000.016, HashMap 702.784
Puts 100.000, size 1.000.000: CompactFloatArray 4.016.472, float[] 4.000.016, HashMap 6.607.808
Puts 1.000.000, size 1.000.000: CompactFloatArray 4.016.472, float[] 4.000.016, HashMap 44.644.032
Puts 1.000, size 5.000.000: CompactFloatArray 1.128.536, float[] 20.000.016, HashMap 72.256
Puts 10.000, size 5.000.000: CompactFloatArray 8.168.536, float[] 20.000.016, HashMap 704.832
Puts 100.000, size 5.000.000: CompactFloatArray 20.013.144, float[] 20.000.016, HashMap 7.385.152
Puts 1.000.000, size 5.000.000: CompactFloatArray 20.131.160, float[] 20.000.016, HashMap 66.395.584
Puts 1.000, size 10.000.000: CompactFloatArray 1.275.992, float[] 40.000.016, HashMap 72.256
Puts 10.000, size 10.000.000: CompactFloatArray 9.289.816, float[] 40.000.016, HashMap 705.280
Puts 100.000, size 10.000.000: CompactFloatArray 37.130.328, float[] 40.000.016, HashMap 7.418.112
Puts 1.000.000, size 10.000.000: CompactFloatArray 40.262.232, float[] 40.000.016, HashMap 69.282.496
{noformat}
I want to share this intermediately, without further interpretation/conclusion for now (I just need to get the train).
> Make external scoring more efficient (ExternalFileField, FileFloatSource)
> -------------------------------------------------------------------------
>
> Key: SOLR-2583
> URL: https://issues.apache.org/jira/browse/SOLR-2583
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Martin Grotzke
> Priority: Minor
> Attachments: FileFloatSource.java.patch, patch.txt
>
>
> External scoring eats much memory, depending on the number of documents in the index. The ExternalFileField (used for external scoring) uses FileFloatSource, where one FileFloatSource is created per external scoring file. FileFloatSource creates a float array with the size of the number of docs (this is also done if the file to load is not found). If there are much less entries in the scoring file than there are number of docs in total the big float array wastes much memory.
> This could be optimized by using a map of doc -> score, so that the map contains as many entries as there are scoring entries in the external file, but not more.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org