You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Adrien Grand (JIRA)" <ji...@apache.org> on 2012/05/24 18:01:29 UTC
[jira] [Created] (SOLR-3486) The memory size of Solr caches should
be configurable
Adrien Grand created SOLR-3486:
----------------------------------
Summary: The memory size of Solr caches should be configurable
Key: SOLR-3486
URL: https://issues.apache.org/jira/browse/SOLR-3486
Project: Solr
Issue Type: Improvement
Components: search
Reporter: Adrien Grand
Priority: Minor
It is currently possible to configure the sizes of Solr caches based on the number of entries of the cache. The problem is that the memory size of cached values may vary a lot over time (depending on IndexReader.maxDoc and the queries that are run) although the JVM heap size does not.
Having a configurable max size in bytes would also help optimize cache utilization, making it possible to store more values provided that they have a small memory footprint.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3486) The memory size of Solr caches
should be configurable
Posted by "Shawn Heisey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283461#comment-13283461 ]
Shawn Heisey commented on SOLR-3486:
------------------------------------
It's going to take me a while to digest what you've just said, but my first thought is that I can't change the implementation without destroying the O(1) nature. The cache is implemented in two parts - a simple map (HashMap) for fast lookup, and an array of sets (LinkedHashSet[]) for fast frequency ordering. When the frequency for an entry needs to be changed, it is removed from one set and added to another.
Although it's not implemented as an actual iterator method, I have code to iterate over the array. I should probably create an iterator and backwards iterator, just to eliminate some duplicate code. If I don't already have a remove method, I should be able to add one.
> The memory size of Solr caches should be configurable
> -----------------------------------------------------
>
> Key: SOLR-3486
> URL: https://issues.apache.org/jira/browse/SOLR-3486
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Adrien Grand
> Priority: Minor
> Attachments: SOLR-3486.patch, SOLR-3486.patch
>
>
> It is currently possible to configure the sizes of Solr caches based on the number of entries of the cache. The problem is that the memory size of cached values may vary a lot over time (depending on IndexReader.maxDoc and the queries that are run) although the JVM heap size does not.
> Having a configurable max size in bytes would also help optimize cache utilization, making it possible to store more values provided that they have a small memory footprint.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3486) The memory size of Solr caches should
be configurable
Posted by "Adrien Grand (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrien Grand updated SOLR-3486:
-------------------------------
Attachment: SOLR-3486.patch
This patch makes the maximum memory size of LRUCache configurable. It would be nice to add the same feature to the concurrent caches although I expect it to be a little more tricky.
> The memory size of Solr caches should be configurable
> -----------------------------------------------------
>
> Key: SOLR-3486
> URL: https://issues.apache.org/jira/browse/SOLR-3486
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Adrien Grand
> Priority: Minor
> Attachments: SOLR-3486.patch
>
>
> It is currently possible to configure the sizes of Solr caches based on the number of entries of the cache. The problem is that the memory size of cached values may vary a lot over time (depending on IndexReader.maxDoc and the queries that are run) although the JVM heap size does not.
> Having a configurable max size in bytes would also help optimize cache utilization, making it possible to store more values provided that they have a small memory footprint.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3486) The memory size of Solr caches should
be configurable
Posted by "Adrien Grand (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrien Grand updated SOLR-3486:
-------------------------------
Attachment: SOLR-3486.patch
Hi Shawn,
I modified the patch in order to make it easier to add this functionality to other cache implementations. All you need to do for SOLR-3393 to support maximum memory size is to split your implementation into a LFU map (a regular map, with no evictions) which iterates (entrySet().iterator()) in frequency order and a LFU cache (that will probably extend or wrap this LFU map). Then to have a LFU cache with a fixed max mem size, just wrap your LFU map into a new SizableCache instance.
> The memory size of Solr caches should be configurable
> -----------------------------------------------------
>
> Key: SOLR-3486
> URL: https://issues.apache.org/jira/browse/SOLR-3486
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Adrien Grand
> Priority: Minor
> Attachments: SOLR-3486.patch, SOLR-3486.patch
>
>
> It is currently possible to configure the sizes of Solr caches based on the number of entries of the cache. The problem is that the memory size of cached values may vary a lot over time (depending on IndexReader.maxDoc and the queries that are run) although the JVM heap size does not.
> Having a configurable max size in bytes would also help optimize cache utilization, making it possible to store more values provided that they have a small memory footprint.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3486) The memory size of Solr caches should
be configurable
Posted by "Adrien Grand (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrien Grand updated SOLR-3486:
-------------------------------
Attachment: LFUMap.java
I've just uploaded LFUMap.java based on your implementation of LFUCache. To have a LFU cache with configurable maximum size in bytes, just wrap an instance of this class into a SizableCache.
I uploaded this file to show how SizableCache could be used with different kinds of backends. But building a LFUCache is a different issue. I think we should continue the discussion on LFUCache on SOLR-3393 and only discuss configurability of the mem size of Solr caches here. Feel free to reuse the code LFUMap.java if you want, just beware that I didn't test it much. :-)
> The memory size of Solr caches should be configurable
> -----------------------------------------------------
>
> Key: SOLR-3486
> URL: https://issues.apache.org/jira/browse/SOLR-3486
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Adrien Grand
> Priority: Minor
> Attachments: LFUMap.java, SOLR-3486.patch, SOLR-3486.patch
>
>
> It is currently possible to configure the sizes of Solr caches based on the number of entries of the cache. The problem is that the memory size of cached values may vary a lot over time (depending on IndexReader.maxDoc and the queries that are run) although the JVM heap size does not.
> Having a configurable max size in bytes would also help optimize cache utilization, making it possible to store more values provided that they have a small memory footprint.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3486) The memory size of Solr caches
should be configurable
Posted by "Shawn Heisey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283142#comment-13283142 ]
Shawn Heisey commented on SOLR-3486:
------------------------------------
This is an awesome idea. SOLR-3393 is a new implementation of LFUCache, I'll have to figure out how to include this, unless you want to give it a try.
> The memory size of Solr caches should be configurable
> -----------------------------------------------------
>
> Key: SOLR-3486
> URL: https://issues.apache.org/jira/browse/SOLR-3486
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Adrien Grand
> Priority: Minor
> Attachments: SOLR-3486.patch
>
>
> It is currently possible to configure the sizes of Solr caches based on the number of entries of the cache. The problem is that the memory size of cached values may vary a lot over time (depending on IndexReader.maxDoc and the queries that are run) although the JVM heap size does not.
> Having a configurable max size in bytes would also help optimize cache utilization, making it possible to store more values provided that they have a small memory footprint.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org