[jira] [Comment Edited] (SOLR-8349) Allow sharing of large in memory data structures across cores

["Gus Heck (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142799#comment-15142799 ] 

Gus Heck edited comment on SOLR-8349 at 2/11/16 2:32 PM:
---------------------------------------------------------

I am not sure I understand what you mean by load the analysis component into the container cache. The analysis component in lucene needs some way to reference the container and ask for the shared resource. My assumption was that it was not ok to add dependencies to solr classes into Lucene. Otherwise I could just cast to SolrResourceLoader in the analysis component in question. Did you look at SOLR-3443 where I use this patch to implement the feature for HunspellStemFilterFactory.java. do this:

{code}
if (loader instanceof ContainerResourceSharing) {
        resourceSharing = (ContainerResourceSharing) loader;
{code}

this then allows me to have a method like:
{code}
  public Dictionary getDictionary() {
    return this.dictionary == null ? (Dictionary) resourceSharing.getContainerResource(resourceKey) : this.dictionary;
  }
{code}
If I can use a Solr Class, that could just as easily say 

{code}
if (loader instanceof SolrResourceLoader) {
        resourceSharing = (SolrResourceLoader) loader;
{code}

One could also add methods to ResourceLoader itself, but then all forms of resource loader have to deal with implementing this method.


was (Author: gus_heck):
I am not sure I understand what you mean by load the analysis component into the container cache. The analysis component in lucene needs some way to reference the container and ask for the shared resource. My assumption was that it was not ok to add dependencies to solr dependencies into Lucene. Otherwise I could just cast to SolrResourceLoader in the analysis component in question. Did you look at SOLR-3443 where I use this patch to implement the feature for HunspellStemFilterFactory.java. do this:

{code}
if (loader instanceof ContainerResourceSharing) {
        resourceSharing = (ContainerResourceSharing) loader;
{code}

this then allows me to have a method like:
{code}
  public Dictionary getDictionary() {
    return this.dictionary == null ? (Dictionary) resourceSharing.getContainerResource(resourceKey) : this.dictionary;
  }
{code}
If I can use a Solr Class, that could just as easily say 

{code}
if (loader instanceof SolrResourceLoader) {
        resourceSharing = (SolrResourceLoader) loader;
{code}

One could also add methods to ResourceLoader itself, but then all forms of resource loader have to deal with implementing this method.

> Allow sharing of large in memory data structures across cores
> -------------------------------------------------------------
>
>                 Key: SOLR-8349
>                 URL: https://issues.apache.org/jira/browse/SOLR-8349
>             Project: Solr
>          Issue Type: Improvement
>          Components: Server
>    Affects Versions: 5.3
>            Reporter: Gus Heck
>         Attachments: SOLR-8349.patch
>
>
> In some cases search components or analysis classes may utilize a large dictionary or other in-memory structure. When multiple cores are loaded with identical configurations utilizing this large in memory structure, each core holds it's own copy in memory. This has been noted in the past and a specific case reported in SOLR-3443. This patch provides a generalized capability, and if accepted, this capability will then be used to fix SOLR-3443.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org