You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Henri Biestro (JIRA)" <ji...@apache.org> on 2007/11/12 12:41:52 UTC
[jira] Created: (SOLR-409) Allow configurable class loader sharing
between cores
Allow configurable class loader sharing between cores
-----------------------------------------------------
Key: SOLR-409
URL: https://issues.apache.org/jira/browse/SOLR-409
Project: Solr
Issue Type: Sub-task
Affects Versions: 1.3
Reporter: Henri Biestro
Priority: Minor
Fix For: 1.3
WHAT:
This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
That element can also specify through the 'shared' attribute whether the library is shared.
By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
Syntax:
<libDir shared='true*|false'>/path/to/shareable/dir</libDir>
WHY:
Current behavior allocates one class loader per config & thus per core.
However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
HOW:
The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
STATUS:
initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-409) Allow configurable class loader
sharing between cores
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12543693 ]
Hoss Man commented on SOLR-409:
-------------------------------
as i mentioned on the mailing list a while back, it's not clear to me how multicore stuff is expected to be used by people in the solr webapp (still hoping to see some use cases / hypothetical-howtos on the wiki) but based on the description/comments in this issue it seems that there are two ways a libDir can be shared...
1) by specifying a libDir in some file i've never seen before called "multicore.xml"
2) by declaring it shared in the solrconfig.xml files of 2 or more cores
#1 makes sense to me ... the correct thing to do seems to be to let whatever new object exists to manage cores (CoreManager?) have it's own ClassLoader that is used when initializing cores (this will then become the parent loader for the class loaders created by each core) ... this "multicore parent classloader" would be configured in the CoreManager's config file (multicore.xml?)
#2 seems scary ... particularly because it allows "sister" cores, initialized by the same classloader to share instances of a class that their own classloader doesn't know about. there's also the issues of how things should behave if two different cores refer to the same physical jar/classfile but do so using different paths (which might be symlinks or hardlinks and won't even have the same canonical path); as well as the question of what happens when we dynamically "unload/reload all of the cores that were sharing a lib -- depending on the order of events, and garbage collection, classes might be completely reloaded (or not)
let's keep the classloader semantics simple, and hierarchical:
1) every core has a plugin classloader.
2) every plugin classloader has a common parent classloader
3) the common parent classloader may contain shared libraries
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by "Henri Biestro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Biestro updated SOLR-409:
-------------------------------
Attachment: solr-350_409.patch
updated to fix typos noted by Walter;
added core name in logging operation where possible (Walter's suggestion);
moved startTime initialization of core after schema parsing (fast machine on reload core was failing test...);
Unified patch from Solaris10 box -
created with: svn diff --diff-cmd /usr/local/bin/diff -x "-w -B -b -E -d -N -u" > ~/solr-350_409.patch
applied with: /usr/local/bin/patch -u -p 0 < ~/solr-350_409.patch
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409_414.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the multicore.xml the parent class loader of all core class loaders used to dynamically create instances.
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The sharedLib attribute is extracted from the XML (multicore.xml) configuration file and parsed in the MultiCore load method. The directory path is used to create an URL class loader that will become the parent class loader of all core class loaders; since class resolution if performed on a parent-first basis, this allows sharing instances between different cores.
> STATUS:
> operational in conjunction with solr-350
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan McKinley resolved SOLR-409.
--------------------------------
Resolution: Fixed
Assignee: Ryan McKinley
This was committed along with SOLR-350
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Assignee: Ryan McKinley
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409_414.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the multicore.xml the parent class loader of all core class loaders used to dynamically create instances.
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The sharedLib attribute is extracted from the XML (multicore.xml) configuration file and parsed in the MultiCore load method. The directory path is used to create an URL class loader that will become the parent class loader of all core class loaders; since class resolution if performed on a parent-first basis, this allows sharing instances between different cores.
> STATUS:
> operational in conjunction with solr-350
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-409) Allow configurable class loader
sharing between cores
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12543970 ]
Hoss Man commented on SOLR-409:
-------------------------------
i'm not sure how to interpret "Yes #1, not #2" ... are you saying my understanding about case #2 is wrong, or that you agree with me we shouldn't do it? (i was basing that comment on the original issue description)
>> I don't see a need to have 'hierarchical' classloaders - a single global shared library should be sufficient:
by hierarchical i'm referring to the normal way java classloaders work: every classloader has a parent, which it delegates to automaticly. if we're going to have a shared class loader available for all cores, then it should be the parent of each core specific plugin classloader.
(currently the plugin ClassLoader uses the context classloader as it's parent ... that can still work as long as whatever "MultiCoreManager" object that deals with multicore.xml uses the new "shared code classloader" as the context when instantiating each SolrCore ... or we can be more explicit about it.)
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by "Henri Biestro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Biestro updated SOLR-409:
-------------------------------
Attachment: solr-350_409.patch
cleaned up, less noisy version on trunk 601457;
updated syntax for shared dir in multicore config;
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409_414.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-409) Allow configurable class loader
sharing between cores
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12544016 ]
Ryan McKinley commented on SOLR-409:
------------------------------------
I agree with everything you say above.
Lets take the approach in #1 and avoid anything resembling #2
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by Walter Ferrara <wa...@gmail.com>.
With a multicore.xml that looks like:
<multicore enabled="true" adminpath="/admin/multicore"
persistent="true" libDir="solr/lib" >
<core name="core0" instanceDir="core0" default="true"/>
<core name="core1" instanceDir="core1" />
</multicore>
it seems to work fine.
Just a tip:
libDir is somehow not relative to configdir. Maybe making it relative to
cfgDir, like:
libDir=cfgdir+libDir;
@ row 83 of MultiCore.java could make it more intuitive. (ie.
libDir="lib" vs. libDir="solr/lib")
I will test it better,
thanks,
Walter
Henri Biestro (JIRA) wrote:
> [ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Henri Biestro updated SOLR-409:
> -------------------------------
>
> Attachment: solr-350_409.patch
>
> solr 350 & solr 409 mixed together
>
>
>> Allow configurable class loader sharing between cores
>> -----------------------------------------------------
>>
>> Key: SOLR-409
>> URL: https://issues.apache.org/jira/browse/SOLR-409
>> Project: Solr
>> Issue Type: Sub-task
>> Affects Versions: 1.3
>> Reporter: Henri Biestro
>> Priority: Minor
>> Fix For: 1.3
>>
>> Attachments: solr-350_409.patch, solr-409.patch, solr-409.patch
>>
>>
>> WHAT:
>> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
>> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
>> That element can also specify through the 'shared' attribute whether the library is shared.
>> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
>> Syntax:
>> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
>> WHY:
>> Current behavior allocates one class loader per config & thus per core.
>> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
>> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
>> HOW:
>> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
>> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
>> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
>> STATUS:
>> initial code drop
>>
>
>
[jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by "Henri Biestro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Biestro updated SOLR-409:
-------------------------------
Attachment: solr-350_409.patch
solr 350 & solr 409 mixed together
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-409) Allow configurable class loader
sharing between cores
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12543787 ]
Ryan McKinley commented on SOLR-409:
------------------------------------
Yes #1, not #2
{code:xml}
<multicore enabled="true" adminpath="/admin/multicore" persistent="true" sharedLib="shared">
<core name="core0" instanceDir="core0" default="true"/>
<core name="core1" instanceDir="core1" />
</multicore>
{code}
I don't see a need to have 'hierarchical' classloaders - a single global shared library should be sufficient:
1) every core has a plugin classloader.
2) MultiCore *may* have a shared classloader
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-409) Allow configurable class loader
sharing between cores
Posted by "Walter Ferrara (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550002 ]
Walter Ferrara commented on SOLR-409:
-------------------------------------
Thanks for committing this patch.
I noticed that when you have just one core (with no multicore.xml), logging says "null", i.e.
INFO: [null] Registered new searcher ....
This could be fixed in several ways:
* by giving a (meaningful) name to the core, when multicore is not used. (like schema.getName())
* by not adding the name of the core when logging is off (which maybe means reuse logStr function, and check if Multicore is enabled)
This is present also in stats.jsp where a null is printed before the uppercase bold "CORE" string.
IMHO, we should set the name of single core when multicore is not set - this may make thing easier; set it as the name of its schema could be a solution.
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409_414.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the multicore.xml the parent class loader of all core class loaders used to dynamically create instances.
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The sharedLib attribute is extracted from the XML (multicore.xml) configuration file and parsed in the MultiCore load method. The directory path is used to create an URL class loader that will become the parent class loader of all core class loaders; since class resolution if performed on a parent-first basis, this allows sharing instances between different cores.
> STATUS:
> operational in conjunction with solr-350
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-409) Allow configurable class loader
sharing between cores
Posted by "Walter Ferrara (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541762 ]
Walter Ferrara commented on SOLR-409:
-------------------------------------
Thanks very much!
a) In order to apply the patch, I've locally modified it to look for Config.java in src/java/org/apache/solr/core/Config.java
b) Don't seems to work with SOLR-350. (I understand it just depends on solr-215, which is in trunk, but what about SOLR-350? It could be interesting to apply it over solr-350, for a future join maybe)
If I apply solr-409.patch before SOLR-350-MultiCore.patch, the 409 goes well and 350 produce errors, otherwise the 350 goes well, and then:
patching file src/java/org/apache/solr/core/Config.java
Hunk #2 FAILED at 54.
Hunk #3 succeeded at 91 (offset 1 line).
Hunk #4 succeeded at 178 (offset 1 line).
Hunk #5 succeeded at 344 (offset 10 lines).
Hunk #6 succeeded at 364 (offset 10 lines).
Hunk #7 succeeded at 405 (offset 10 lines).
Hunk #8 succeeded at 424 (offset 10 lines).
1 out of 8 hunks FAILED -- saving rejects to file src/java/org/apache/solr/core/Config.java.rej
ALSO:
We should choose where is the best place to put the info about the shared lib dir.
In this patch I can choose if to share or not to share, which is ok for me, but an user could be so lazy that wants to share just something, not every jars of his core :-)
Another approach could be to use SOLR-350 and put the info about the share directory info in multicore.xml, like:
<multicore enabled="true" adminpath="/admin/multicore" persistent="true" sharedLibDir="lib">
<core name="core0" instanceDir="core0" default="true"/>
<core name="core1" instanceDir="core1" />
</multicore>
Where sharedLibDir indicate where to locate shared libraries - this way it could be the same of a core, but it could be a totally different one too.
Any thoughts?
--
Walter
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by Walter Ferrara <wa...@gmail.com>.
some (irrelevant) typos in MultiCore:
//[row 55] private SolrCore defaulCore = null;
[row 123] "multicore.xml defiens multiple default cores. "+
[row 210] public void setDefautlCore( SolrCore core )
[row 270] public SolrCore getDefaulCore()
Great job!,
Walter
Ryan McKinley (JIRA) wrote:
> [ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Ryan McKinley updated SOLR-409:
> -------------------------------
>
> Attachment: solr-350_409.patch
>
> This adds solrj support for multicore and uses it to run a bunch of tests. The tests work via jetty and embedded.
>
> Currently the supported "actions" are "status", " SetAsDefault", and "reload". Persistence and dynamic core creation are still down the line.
>
> This patch should have no impact on standard single core access. This adds the multicore functionality to setups that have a 'multicore.xml' file in the 'solr.solr.home' directory.
>
> Without objection, I will commit this in the next day or two.
>
>
>> Allow configurable class loader sharing between cores
>> -----------------------------------------------------
>>
>> Key: SOLR-409
>> URL: https://issues.apache.org/jira/browse/SOLR-409
>> Project: Solr
>> Issue Type: Sub-task
>> Affects Versions: 1.3
>> Reporter: Henri Biestro
>> Priority: Minor
>> Fix For: 1.3
>>
>> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409_414.patch, solr-409.patch, solr-409.patch
>>
>>
>> WHAT:
>> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
>> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
>> That element can also specify through the 'shared' attribute whether the library is shared.
>> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
>> Syntax:
>> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
>> WHY:
>> Current behavior allocates one class loader per config & thus per core.
>> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
>> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
>> HOW:
>> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
>> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
>> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
>> STATUS:
>> initial code drop
>>
>
>
[jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan McKinley updated SOLR-409:
-------------------------------
Attachment: solr-350_409.patch
This adds solrj support for multicore and uses it to run a bunch of tests. The tests work via jetty and embedded.
Currently the supported "actions" are "status", " SetAsDefault", and "reload". Persistence and dynamic core creation are still down the line.
This patch should have no impact on standard single core access. This adds the multicore functionality to setups that have a 'multicore.xml' file in the 'solr.solr.home' directory.
Without objection, I will commit this in the next day or two.
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409_414.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by "Henri Biestro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Biestro updated SOLR-409:
-------------------------------
Attachment: solr-409.patch
initial drop
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-409) Allow configurable class loader
sharing between cores
Posted by "Walter Ferrara (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12542562 ]
Walter Ferrara commented on SOLR-409:
-------------------------------------
I ran some test, it seems to work really fine.
I like the way you choose to show admin pages for each cores.
I noticed that when I click on "SCHEMA" I get:
<ERROR>
Permission denied for file solr\core0//conf/schema.xml
</ERROR>
"SCHEMA" link point to http://localhost:8983/solr/admin/get-file.jsp?core=core0&file=solr\core0//conf/schema.xml
Interesting "CONFIG" links works fine (http://localhost:8983/solr/admin/get-file.jsp?core=core0&file=solrconfig.xml)
taking a look at index.jsp in admin seems that core.getSchemaFile() add paths, while core.getConfigFile() don't.
I had an idea:
Solr currently log info to the "console", this info have no track of which core executed the action, e.g.
14-nov-2007 20.33.51 org.apache.solr.core.SolrCore execute
INFO: /select/ indent=on&start=0&q=solr&version=2.2&rows=10 0 0
14-nov-2007 20.33.54 org.apache.solr.core.SolrCore execute
INFO: /select/ indent=on&start=0&q=solr&version=2.2&rows=10 0 0
The first one was executed on "core0", while the latter on "core1" - no way to determine it from those log lines.
I was wondering, what are your thoughts about putting core name somewhere in logging info, ie.
14-nov-2007 20.33.51 org.apache.solr.core.SolrCore execute
INFO: [core0] /select/ indent=on&start=0&q=solr&version=2.2&rows=10 0 0
14-nov-2007 20.33.54 org.apache.solr.core.SolrCore execute
INFO: [core1] /select/ indent=on&start=0&q=solr&version=2.2&rows=10 0 0
--
Walter
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by "Henri Biestro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Biestro updated SOLR-409:
-------------------------------
Attachment: solr-350_409.patch
lib dir is now relative to multicore dir;
started servlet update for solr-350 enhancements (core as http parameter)
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by "Henri Biestro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Biestro updated SOLR-409:
-------------------------------
Attachment: solr-350_409.patch
admin webapp is core aware
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by "Henri Biestro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Biestro updated SOLR-409:
-------------------------------
Attachment: solr-409.patch
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by "Henri Biestro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Biestro updated SOLR-409:
-------------------------------
Description:
WHAT:
This patch allows to configure in the multicore.xml the parent class loader of all core class loaders used to dynamically create instances.
WHY:
Current behavior allocates one class loader per config & thus per core.
However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
HOW:
The sharedLib attribute is extracted from the XML (multicore.xml) configuration file and parsed in the MultiCore load method. The directory path is used to create an URL class loader that will become the parent class loader of all core class loaders; since class resolution if performed on a parent-first basis, this allows sharing instances between different cores.
STATUS:
operational in conjunction with solr-350
was:
WHAT:
This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
That element can also specify through the 'shared' attribute whether the library is shared.
By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
Syntax:
<libDir shared='true*|false'>/path/to/shareable/dir</libDir>
WHY:
Current behavior allocates one class loader per config & thus per core.
However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
HOW:
The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
STATUS:
initial code drop
updated description to reflect decisions & match implementation
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409_414.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the multicore.xml the parent class loader of all core class loaders used to dynamically create instances.
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The sharedLib attribute is extracted from the XML (multicore.xml) configuration file and parsed in the MultiCore load method. The directory path is used to create an URL class loader that will become the parent class loader of all core class loaders; since class resolution if performed on a parent-first basis, this allows sharing instances between different cores.
> STATUS:
> operational in conjunction with solr-350
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-409) Allow configurable class loader sharing
between cores
Posted by "Henri Biestro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Biestro updated SOLR-409:
-------------------------------
Attachment: solr-350_409_414.patch
experimental; merges solr-350 + 414 + 409 (feature matches comments made)
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-350_409.patch, solr-350_409.patch, solr-350_409.patch, solr-350_409_414.patch, solr-409.patch, solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-409) Allow configurable class loader
sharing between cores
Posted by "Henri Biestro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541771 ]
Henri Biestro commented on SOLR-409:
------------------------------------
Sorry about the non-regular patch, my bad, I should've documented that.
I like the idea of being able to specify a default shared lib directory for all cores in the multicore configuration.
I've thus modified Config.java to allow overriding how to solve the default shared library (I'm producing the patch the same way as the first one, so to apply it on a trunk, you may want to 'cd src/java/org/.apache/solr/core' first).
>From a solr-350 base, I've updated MultiCore to look for a 'libDir' attribute that serves as the default shared library.
To ease testing, I'm thus also uploading the solr-350 + solr-409 patch here.
> Allow configurable class loader sharing between cores
> -----------------------------------------------------
>
> Key: SOLR-409
> URL: https://issues.apache.org/jira/browse/SOLR-409
> Project: Solr
> Issue Type: Sub-task
> Affects Versions: 1.3
> Reporter: Henri Biestro
> Priority: Minor
> Fix For: 1.3
>
> Attachments: solr-409.patch
>
>
> WHAT:
> This patch allows to configure in the solrconfig.xml the library directory and associated class loader used to dynamically create instances.
> The solr config XML can now specify a libDir element (the same way the dataDir can be specified).
> That element can also specify through the 'shared' attribute whether the library is shared.
> By default, the shared attribute is true; if you specify a libDir, its class loader is made shareable.
> Syntax:
> <libDir shared='true*|false'>/path/to/shareable/dir</libDir>
> WHY:
> Current behavior allocates one class loader per config & thus per core.
> However, there are cases where one would like different cores to share some objects that are dynamically instantiated (ie, where the class name is used to find the class through the class loader and instantiate). In the current form; since each core possesses its own class loader, static members are indeed different objects. For instance, there is no way of implementing a singleton shared between 2 request handlers.
> Originally from http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html
> HOW:
> The libDir element is extracted from the XML configuration file and parsed in the Config constructor.
> A static map of weak reference to classloaders keyed by library path is used to keep track of already built shareable class loaders.
> The weak reference is here to allow class eviction by the GC, avoiding leakage that would result by keeping hard reference to class loaders.
> STATUS:
> initial code drop
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.