Post-SOLR215/SOLR350 singleton issue

[Walter Ferrara <wa...@gmail.com>]

I'm currently using latest solr + SOLR350 in a multicore solr.
I've a ConnectionPool, which use a singleton design, it has been tested
and it really works as singleton.
Every cores have an handler, responsible for update that will eventually
use that connection pool.

The problem is: every core re-instance the connectionpool (this seems to
happen with a test singleton too), like what would happen if every core
reside in a different JVMs. This happens when the handler is called (not
inited), at different time period; inside the same core, it work as
singleton, but others cores, when called, will simple make a new ones.

Is there a way to bypass that, and to use singletons in a "singleton"
fashion between cores?
Btw: Cores do see MultiCore as a singleton, i.e. all handler see the
same object, so I could get around this problem using MultiCore to get
cores..., but I would like to understand why cores doesn't seems to
share singletons. I may be wrong, but this used to work with (old)
Henri's solr215 patch.

Thanks,
Walter

Re: Post-SOLR215/SOLR350 singleton issue

[Chris Hostetter <ho...@fucit.org>]

: My apologies for generating confusion.

oh please don't ... some confusion is bound to be a by-product of active 
experimentation and forward thinking change ... we just need to make sure 
that we periodicly take stock, and double check that we know where we are 
going.

: Hopes this makes the mess more understandable.
: Any other advice besides the kind suggestions you already made?

Well, I still think it would be really useful to have some 
non-jira-comment docs that describe how we we expect things to work in the 
future ... sample uses cases showing the types of things that will be 
possible, descriptions of (new) config syntax in those use cases (didn't i 
see something somewhere about a new "multicore.xml" file?), and a 
description of what will go on under the covers.

Ultimately these docs will need to be written anyway to serve as "user 
guides" -- if we right them now they also serve as "blue prints" for the 
user experience.


-Hoss

Re: Post-SOLR215/SOLR350 singleton issue

[Henrib <hb...@gmail.com>]

My apologies for generating confusion.

The original solr-215 has a small blurb about its intentions that still
remain true today that I'll try to beef up and make more comprehensible (in
solr-350 most probably). There are 2 functional axis; one axis is about
easing deployment in IT constrained environments -with independent cores but
only one web-application- and the other one is to allow cooperative indexes
- document collections with different life-cycles (localization for instance
where one index might not be the best fit) or embedding multiple cores in a
functionally wider application.

When solr-215 was committed and solr-350 created, the solr-215 code that
relied on a static for the class loader was cleaned up and each core was
granted its own class loader. The recent activity started after Walter hit
an issue with these multiple class loaders ; this introduced a lot of
complexity to share objects between handlers (for instance). Thus, solr-409
(sub-task of solr-215) which is just a solution to that problem, allowing to
specify whether a class loader should be shared between 2 core instances.
This incidentally reflects the functional needs of the 2 orthogonal axes
previously defined.

On the plug-in side, solr-399 is meant to allow 'core initialization' of
plug-ins: if such a plug-in shares some information across its different
instances, one could use this feature to perform per-core injection on that
shared object. So even if we share an object through a plug-in, we'd still
be able to let it know that different cores are accessing it. I realize
mentioning this was unfortunately confusing... 

As for the administration of multiple cores, aka solr-350, I've restarted
working on it - but got sidetracked by solr-399 & solr-409 (not to mention
my interest in solr-281). There is even more work after to tackle
replication...

Hopes this makes the mess more understandable.
Any other advice besides the kind suggestions you already made?


-- 
View this message in context: http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html#a13710603
Sent from the Solr - Dev mailing list archive at Nabble.com.

Re: Post-SOLR215/SOLR350 singleton issue

[Chris Hostetter <ho...@fucit.org>]

: Created solr-409 as a sub-task of solr-215 to track and post patch.
: I took the shareable library directory path (ie you can specify the library
: directory used by a given core and whether it is shared or not); if 2 cores
: specify the same directory and they are shared, there will be only one
: instance of the class loader.

This "lib: shared or not" issue seems like just one specific question in a 
larger issue of what the long term plan is.

I admit i haven't been following all the multi-core stuff all that 
closely, i know the first goal was to get support on to the internal APIs 
(done) and the next goal is to add support into the WAR for managing 
cores, but i'm completely lost as to what exactly that is going to mean 
... in some discussions it seems that we're talking about multiple 
isolated indexes in a single webapp; in others there is mention of plugins 
knowing about multiple cores, presumably to be able search multiple 
indexes; and then we have the orthoginal issues of dynamicly modifing the 
schema/configs of a core, and possibly persisting them; reloading cores 
to reinitialize them with new configs; cloning cores to then modify their 
configs; etc...

it all seems a little confusing.

Maybe the heavy hitters involved all have a shared collective vision of 
what things will look like based on past discussions, but it would be nice 
to see that vision written up on the wiki (as some use cases and a high 
level design) so we can verify that every one does in fact have a common 
vision, and so people on the periphery of the big happenings (like me!) 
can understand where we're going.



-Hoss

Re: Post-SOLR215/SOLR350 singleton issue

[Henrib <hb...@gmail.com>]

Created solr-409 as a sub-task of solr-215 to track and post patch.
I took the shareable library directory path (ie you can specify the library
directory used by a given core and whether it is shared or not); if 2 cores
specify the same directory and they are shared, there will be only one
instance of the class loader.
Comments, suggestions welcome.
-- 
View this message in context: http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html#a13704209
Sent from the Solr - Dev mailing list archive at Nabble.com.

Re: Post-SOLR215/SOLR350 singleton issue

[Yonik Seeley <yo...@apache.org>]

On Nov 11, 2007 5:51 AM, Henrib <hb...@gmail.com> wrote:
> It seems to me multicore should avoid replicating (or restricting itself to)
> features what we can get with multiple web apps; I advocate allowing tighter
> couplings between cores in the former which seems to me closer to user
> expectations (at least Walter's).

+1
Let's avoid re-implementing any more of a servlet container than we have to ;-)

-Yonik

Re: Post-SOLR215/SOLR350 singleton issue

[Ryan McKinley <ry...@gmail.com>]

I agree we don't want to replicate the servlet container.

> 
> If we were to create class loaders not per instance but per 'lib' directory
> (not per instance dir as I mentioned earlier but per library directory),

what about the very standard use case where I want to 'reload' a core? 
In this case, i have a single instance directory (and lib directory) and 
want to reload the project so it is in the same state as if you started 
from zero.


> 
> I believe we can keep core strict independence *and* add sharing core
> functionality allowing a wider choice to answer users needs.
> 

I don't quite follow yet, but I agree with the goal.


ryan

Re: Post-SOLR215/SOLR350 singleton issue

[Walter Ferrara <wa...@gmail.com>]

Thanks for your answers.
I've made some test, the singleton is correctly shared between handlers
in the same core, but not between cores, while putting connection pool
jar (and its dependencies) inside the war, make it works as it should
(but that doesn't look like a solution to me).
It seems to me that this behavior is perfectly in line with what
discussed in this thread.

In my opinion, Henrib's approach (one "shared" lib, and one "private"
lib for each core) would meet my needs, and also allow that "separation"
between cores, you are aiming at; this way it would be up to the users
to choose how to deal with multiple cores, according to their needs, and
the limitations an approach could carry compared to the other.

Thanks,
Walter

Henrib wrote:
> It seems to me multicore should avoid replicating (or restricting itself to)
> features what we can get with multiple web apps; I advocate allowing tighter
> couplings between cores in the former which seems to me closer to user
> expectations (at least Walter's). If we keep one class loader per core
> instance, sharing objects is as 'hard' with multiple cores as it is with
> multiple web apps.
>
> If we were to create class loaders not per instance but per 'lib' directory
> (not per instance dir as I mentioned earlier but per library directory),
> this would allow keeping the choice (make different lib directories for
> classes/instances you dont want to share, share one otherwise). In essence,
> this makes a shareable context named by library dir and implemented through
> class loaders. Or could have a 'shared' library dir in the solr installation
> - and a dedicated class loader (parent-first resolution...).
>
> And since we are also looking at ways to post-init some dynamically created
> objects with a core (solr-399), there will be soon new ways to inject per
> core behavior in shared objects - keeping them segregated if needed being
> one.
>
> I believe we can keep core strict independence *and* add sharing core
> functionality allowing a wider choice to answer users needs.
>
>
>   

Re: Post-SOLR215/SOLR350 singleton issue

[Henrib <hb...@gmail.com>]

It seems to me multicore should avoid replicating (or restricting itself to)
features what we can get with multiple web apps; I advocate allowing tighter
couplings between cores in the former which seems to me closer to user
expectations (at least Walter's). If we keep one class loader per core
instance, sharing objects is as 'hard' with multiple cores as it is with
multiple web apps.

If we were to create class loaders not per instance but per 'lib' directory
(not per instance dir as I mentioned earlier but per library directory),
this would allow keeping the choice (make different lib directories for
classes/instances you dont want to share, share one otherwise). In essence,
this makes a shareable context named by library dir and implemented through
class loaders. Or could have a 'shared' library dir in the solr installation
- and a dedicated class loader (parent-first resolution...).

And since we are also looking at ways to post-init some dynamically created
objects with a core (solr-399), there will be soon new ways to inject per
core behavior in shared objects - keeping them segregated if needed being
one.

I believe we can keep core strict independence *and* add sharing core
functionality allowing a wider choice to answer users needs.


-- 
View this message in context: http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html#a13690225
Sent from the Solr - Dev mailing list archive at Nabble.com.

Re: Post-SOLR215/SOLR350 singleton issue

[Ryan McKinley <ry...@gmail.com>]

> 
> perhaps.  it depends largely on what the long term goals of multi-core
> support are ... if we're striving for "dynamicly" creating "solr contexts" 
> independently of "servlet contexts" then the current behavior is probably 
> ideal ... we're protecting the plugins of each SolrCore from 
> corrupting eachother.

agreed.  In the current setup, the only classes shared across all 
SolrCore instances are the RequestDispatcher and the MultiCore registry. 
  If you reload a core, all the static initialization stuff should also 
be reloaded.

*perhaps* adding a 'context' to the MultiCore registry would be a good 
way to let cores communicate with one another.

ryan

Re: Post-SOLR215/SOLR350 singleton issue

[Chris Hostetter <ho...@fucit.org>]

: I don't know if this applies in your case but there is now one class loader
: per solr core instance (more exactly per config; it used to be a static 
: before).

: If your connection pool is a static member of a core handler, each core will
: create the handler through different class loaders - thus different
: instances of the static member.

i didn't understand the orriginal issue, but based on Ryan's reply I think 
Henrib is right on the money.

: Internally, it could be better (aka easier to use) to create one class
: loader per instance dir instead of per core.

perhaps.  it depends largely on what the long term goals of multi-core
support are ... if we're striving for "dynamicly" creating "solr contexts" 
independently of "servlet contexts" then the current behavior is probably 
ideal ... we're protecting the plugins of each SolrCore from 
corrupting eachother.

if however we want SolrCores to be able to "communicate" and interact with 
eachother ... well, then a class loader per solr webapp mmight make sense.

honestly: i'm not sure i understand what "instance dir" means in a 
multi-core world ... do all cores share an instance dir?  how do configs 
and dataDirs work in that case?


-Hoss

Re: Post-SOLR215/SOLR350 singleton issue

[Henrib <hb...@gmail.com>]

I don't know if this applies in your case but there is now one class loader
per solr core instance (more exactly per config; it used to be a static 
before).
If your connection pool is a static member of a core handler, each core will
create the handler through different class loaders - thus different
instances of the static member.
Which raises the question of "how do I share objects between handlers
created by different cores then?" - and I dont have an answer to that yet...
Internally, it could be better (aka easier to use) to create one class
loader per instance dir instead of per core.
Hope this helps
Henrib


Walter Ferrara-2 wrote:
> 
> I'm currently using latest solr + SOLR350 in a multicore solr.
> I've a ConnectionPool, which use a singleton design, it has been tested
> and it really works as singleton.
> Every cores have an handler, responsible for update that will eventually
> use that connection pool.
> 
> The problem is: every core re-instance the connectionpool (this seems to
> happen with a test singleton too), like what would happen if every core
> reside in a different JVMs. This happens when the handler is called (not
> inited), at different time period; inside the same core, it work as
> singleton, but others cores, when called, will simple make a new ones.
> 
> Is there a way to bypass that, and to use singletons in a "singleton"
> fashion between cores?
> Btw: Cores do see MultiCore as a singleton, i.e. all handler see the
> same object, so I could get around this problem using MultiCore to get
> cores..., but I would like to understand why cores doesn't seems to
> share singletons. I may be wrong, but this used to work with (old)
> Henri's solr215 patch.
> 
> Thanks,
> Walter
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Post-SOLR215-SOLR350-singleton-issue-tf4776980.html#a13680116
Sent from the Solr - Dev mailing list archive at Nabble.com.

Re: Post-SOLR215/SOLR350 singleton issue

[Ryan McKinley <ry...@gmail.com>]

> 
> It is not related to the singleton, AFAIK, as I tested it with a simple
> multhread app, and I always got the object created one time
> (singletonObject = new getInstance()), while other calls just receive
> the already created object.
> 
> While if the handler try to call getInstance(), the first time (the
> first time /for each core/, and here is the point), init a brand new
> object (other call to getInstance() inside the same core works fine)
> 

So if I understand correctly, you are saying that the singleton object 
within RequestHandler instance for a given SolrCore is *not* shared 
across cores.

What happens if you register multiple RequestHandlers within the same 
core using the same RequestHandler class?  do they share the same 
singleton instance?

If so, I'd suggest looking into class loader issues -- does anything 
change if you bundle your custom classes in the .war file? -- but that 
seems like a stretch.

ryan

Re: Post-SOLR215/SOLR350 singleton issue

[Walter Ferrara <wa...@gmail.com>]

Uhmm I try to explain better, basically is a strange behavior I fighting
with.

I've got a singleton.
Every core have an handler that try to obtain the object by going to
that singleton getInstance() method, which look like this:

public synchronized static SingletonObjectDemo getInstance(){
    if (singletonObject == null){
        singletonObject = new SingletonObjectDemo ();
    }  
    return singletonObject;
}

However every one obtain a different object.

It is not related to the singleton, AFAIK, as I tested it with a simple
multhread app, and I always got the object created one time
(singletonObject = new getInstance()), while other calls just receive
the already created object.

While if the handler try to call getInstance(), the first time (the
first time /for each core/, and here is the point), init a brand new
object (other call to getInstance() inside the same core works fine)

Ryan McKinley wrote:
>
>>
>> Is there a way to bypass that, and to use singletons in a "singleton"
>> fashion between cores?
>> Btw: Cores do see MultiCore as a singleton, i.e. all handler see the
>> same object, so I could get around this problem using MultiCore to get
>> cores..., but I would like to understand why cores doesn't seems to
>> share singletons. I may be wrong, but this used to work with (old)
>> Henri's solr215 patch.
>>
>
> I don't totally understand what issue you are having.  What is the
> specific function call that is not behaving how you would hope?
>
> Is it that: private static SolrCore instance; is not really a
> singleton?  and that SolrCore.getSolrCore() returns the last SolrCore
> initalized rather then the first one?
>
> My thinking with this is that the 'instance' variable is @Deprecated
> and should go away (along with getSolrCore).  Any requests to get a
> SolrCore should be made with the real singleton class MultiCore --
> that seems cleaner then adding a static registry to SolrCore.
>
> Is that what you are refering to?
>
> ryan
>
>

Re: Post-SOLR215/SOLR350 singleton issue

[Ryan McKinley <ry...@gmail.com>]

> 
> Is there a way to bypass that, and to use singletons in a "singleton"
> fashion between cores?
> Btw: Cores do see MultiCore as a singleton, i.e. all handler see the
> same object, so I could get around this problem using MultiCore to get
> cores..., but I would like to understand why cores doesn't seems to
> share singletons. I may be wrong, but this used to work with (old)
> Henri's solr215 patch.
> 

I don't totally understand what issue you are having.  What is the 
specific function call that is not behaving how you would hope?

Is it that: private static SolrCore instance; is not really a singleton? 
  and that SolrCore.getSolrCore() returns the last SolrCore initalized 
rather then the first one?

My thinking with this is that the 'instance' variable is @Deprecated and 
should go away (along with getSolrCore).  Any requests to get a SolrCore 
should be made with the real singleton class MultiCore -- that seems 
cleaner then adding a static registry to SolrCore.

Is that what you are refering to?

ryan