Embedding Solr vs Lucene, multiple Solr cores?

[Henrib <hb...@gmail.com>]

I'm trying to choose between embedding Lucene versus embedding Solr in one
webapp.

In Solr terms, functional requirements would more or less lead to multiple
schema & conf (need CRUD/generation on those) and deployment constraints
imply one webapp instance. The choice I'm trying to make is thus:
-Embed Lucene and (attempt to) recode a lot of what Solr provides... (the
straw man)
-Embed Solr but refactor 'some' of its core, assuming it is correct to see
one Solr core as the association of one schema & one conf.

There have been a few threads about multiple indexes and/or
multiple/reloading schemas.
>From what I gathered, one solution stems from the 'multiple webapp instances
deployment' and implies 'extracting' the static instance (at least the
SolrCore) & thus host multiple Solr cores in one webapp.

Obviously, the operations (queries/add/delete doc) would need to carry which
core they are targeting (one 'core' being set as the 'default' for
compatibility purpose).
What will be the other big hurdles, the ones that could even preclude the
very idea ? (caches handling, updater threads, HA features...)
Are there any easier routes (class-loaders, 'provisional' schema...) ?

Any advice welcome. Thanks.
Henri


-- 
View this message in context: http://www.nabble.com/Embedding-Solr-vs-Lucene%2C-multiple-Solr-cores--tf3572324.html#a9981355
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Embedding Solr vs Lucene, multiple Solr cores?

[Henrib <hb...@gmail.com>]

I suppose I'm not the only one having to cope with the kind of policies I
was describing (& their idiosynchrasies); in some organizations, trying to
get IT to modify anything related to 'deployment policy' is just (very close
to) impossible... Within those, having a dedicated Tomcat to run the
application is the best thing that can happen! But even in this case, some
will impose only one web application deployed in it... 

Anyway, I've been looking deeper in removing the various static refs &
singletons.
The SolrCore singleton is the easy one but the various Configs are a bit
more tedious.
Would you rather have:
a/ 1 'config' package to gather all those configuration objects
(Query/Index/Admin/Cache/etc...)
b/ generalize the SolrIndexConfig kind, one SolrxxxConfig per configuration
sensitive class
Would it be ok to refactor the SolrConfig to aggregate those and have one
instance of SolrConfig per SolrCore? 
I lack understanding about JMX though to see how the different SolrCore
instances would be exposed as MBeans (if SolrInfoMBean end up being used
this way); any hint would be appreciated.

Henri


Chris Hostetter wrote:
> 
> 
> : but does not want to deal with the application itself, leaving the
> 'business
> : users' side administer it. Even if there is a dedicated Tomcat for the
> main
> : app, IT will not let the 'business users' install other applications
> (scope
> : of responsibility, code versus data, validation procedures, etc). Thus
> the
> : 'one application' constraint.
> 
> There tends to be a lot of devils in the details of policy discussions
> like this, but perhaps you could redefine the definition of an
> "application" from your ops/biz standpoint to be broader then the
> definition from a servlet container standpoint (ie: let the "application"
> be the entire Tomcat setup running several webapps)
> 
> Alternately, I've heard people mention in past discussions issues
> regarding service provider run servlet containers with self serve WAR hot
> deployment and the issues involved with only being able to hange your WAR
> and not having any control over hte container itself and i've always
> wondered: how hard would be to wrap tomcat (or jetty) so that it is a war
> that can run inside of another servlet container ... then you can have
> multiple wars embeded in that war and control the tomcat configsto your
> hearts content -- treating the ISPs servlet container like an OS.
> 
> : For the sake of argument :-), besides the SolrCore singletons which is
> easy
> : to circumvent (a map of cores & at least a pointer from the instantiated
> : schema to the core handling it, are there others that are hiding
> : (Config.config, caches...) that would preclude the multiple core track?
> 
> There are lots of places in the code where class instances use static refs
> to find the Core/Config/IndexSchema which would have to know know about
> your Map and keys ... it would be a lot of non trivial changes and
> refactoring i believe.
> 
> That said: If anyone is interested in tackling a patch to eliminate all of
> the static Singletons i (and many others i suspect) would be
> extremely gratefull .. both for how much it would improve hte reusability
> of Solr in embedded situatiosn like this, but also for how it would
> (hopefully) make hte code eaier to follow for future developers.
> 
> 
> -Hoss
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Embedding-Solr-vs-Lucene%2C-multiple-Solr-cores--tf3572324.html#a10020669
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Embedding Solr vs Lucene, multiple Solr cores?

[Chris Hostetter <ho...@fucit.org>]

: but does not want to deal with the application itself, leaving the 'business
: users' side administer it. Even if there is a dedicated Tomcat for the main
: app, IT will not let the 'business users' install other applications (scope
: of responsibility, code versus data, validation procedures, etc). Thus the
: 'one application' constraint.

There tends to be a lot of devils in the details of policy discussions
like this, but perhaps you could redefine the definition of an
"application" from your ops/biz standpoint to be broader then the
definition from a servlet container standpoint (ie: let the "application"
be the entire Tomcat setup running several webapps)

Alternately, I've heard people mention in past discussions issues
regarding service provider run servlet containers with self serve WAR hot
deployment and the issues involved with only being able to hange your WAR
and not having any control over hte container itself and i've always
wondered: how hard would be to wrap tomcat (or jetty) so that it is a war
that can run inside of another servlet container ... then you can have
multiple wars embeded in that war and control the tomcat configsto your
hearts content -- treating the ISPs servlet container like an OS.

: For the sake of argument :-), besides the SolrCore singletons which is easy
: to circumvent (a map of cores & at least a pointer from the instantiated
: schema to the core handling it, are there others that are hiding
: (Config.config, caches...) that would preclude the multiple core track?

There are lots of places in the code where class instances use static refs
to find the Core/Config/IndexSchema which would have to know know about
your Map and keys ... it would be a lot of non trivial changes and
refactoring i believe.

That said: If anyone is interested in tackling a patch to eliminate all of
the static Singletons i (and many others i suspect) would be
extremely gratefull .. both for how much it would improve hte reusability
of Solr in embedded situatiosn like this, but also for how it would
(hopefully) make hte code eaier to follow for future developers.


-Hoss

Re: Embedding Solr vs Lucene, multiple Solr cores?

[Henrib <hb...@gmail.com>]

Thank you both for your quick answers.

The one webapp constraint comes from the main 'embedding' application so I
don't have much leeway there. The direct approach was to map the
main/hosting application document collection & types to one schema/conf.
Since the host collections & types can be dynamically created, this seemed
the natural route (albeit hard).

The longer story is that in our typical customer environments, IT deploys &
monitors webapps (provision space & al, replicate for disaster recovery etc)
but does not want to deal with the application itself, leaving the 'business
users' side administer it. Even if there is a dedicated Tomcat for the main
app, IT will not let the 'business users' install other applications (scope
of responsibility, code versus data, validation procedures, etc). Thus the
'one application' constraint.

Anyway, it seems a 'provisional' schema where most fields would be dynamic
and some notational convention to map them would be the easiest route. And
replace the targeted different indexes by equivalent filters. I gather from
your inputs the potential functionality loss and/or performance hit is not
something I should be afraid of.

For the sake of completeness, instead of embedding Solr in that single
instance, I thought about using several Solr instances running in different
webapp instances & use them as 'coprocessors' for the main application; this
would imply serializing/deserializing/redirecting queries & results between
webapps which is not the most efficient way on a single host/VM env (may be
Tomcat crosscontext could help alleviate that). But this would also require
dynamically deploying webapps for that purpose which is a no-no from IT...

For the sake of argument :-), besides the SolrCore singletons which is easy
to circumvent (a map of cores & at least a pointer from the instantiated
schema to the core handling it, are there others that are hiding
(Config.config, caches...) that would preclude the multiple core track?

Thanks again
Henri


Tom Hill-6 wrote:
> 
> Hi -
> 
> Of the various approaches that you could take, the one I'd work on first
> is:
> 
>> deployment constraints imply one webapp instance.
> 
> In most environments, it's going to cost a lot less to change this, than
> to
> try to roll your own, or extensively modify solr.
> 
> I know I'm sidestepping your stated requirements, but I'd take a long look
> at that one.
> 
> BTW, We cut over from an embedded Lucene instance to Solr about 4 months
> ago, and are very happy that we did.
> 
> Tom
> 
> On 4/13/07, Henrib <hb...@gmail.com> wrote:
>>
>>
>> I'm trying to choose between embedding Lucene versus embedding Solr in
>> one
>> webapp.
>>
>> In Solr terms, functional requirements would more or less lead to
>> multiple
>> schema & conf (need CRUD/generation on those) and deployment constraints
>> imply one webapp instance. The choice I'm trying to make is thus:
>> -Embed Lucene and (attempt to) recode a lot of what Solr provides... (the
>> straw man)
>> -Embed Solr but refactor 'some' of its core, assuming it is correct to
>> see
>> one Solr core as the association of one schema & one conf.
>>
>> There have been a few threads about multiple indexes and/or
>> multiple/reloading schemas.
>> From what I gathered, one solution stems from the 'multiple webapp
>> instances
>> deployment' and implies 'extracting' the static instance (at least the
>> SolrCore) & thus host multiple Solr cores in one webapp.
>>
>> Obviously, the operations (queries/add/delete doc) would need to carry
>> which
>> core they are targeting (one 'core' being set as the 'default' for
>> compatibility purpose).
>> What will be the other big hurdles, the ones that could even preclude the
>> very idea ? (caches handling, updater threads, HA features...)
>> Are there any easier routes (class-loaders, 'provisional' schema...) ?
>>
>> Any advice welcome. Thanks.
>> Henri
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Embedding-Solr-vs-Lucene%2C-multiple-Solr-cores--tf3572324.html#a9981355
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: http://www.nabble.com/Embedding-Solr-vs-Lucene%2C-multiple-Solr-cores--tf3572324.html#a9986289
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Embedding Solr vs Lucene, multiple Solr cores?

[Tom Hill <so...@zvents.com>]

Hi -

Of the various approaches that you could take, the one I'd work on first is:

> deployment constraints imply one webapp instance.

In most environments, it's going to cost a lot less to change this, than to
try to roll your own, or extensively modify solr.

I know I'm sidestepping your stated requirements, but I'd take a long look
at that one.

BTW, We cut over from an embedded Lucene instance to Solr about 4 months
ago, and are very happy that we did.

Tom

On 4/13/07, Henrib <hb...@gmail.com> wrote:
>
>
> I'm trying to choose between embedding Lucene versus embedding Solr in one
> webapp.
>
> In Solr terms, functional requirements would more or less lead to multiple
> schema & conf (need CRUD/generation on those) and deployment constraints
> imply one webapp instance. The choice I'm trying to make is thus:
> -Embed Lucene and (attempt to) recode a lot of what Solr provides... (the
> straw man)
> -Embed Solr but refactor 'some' of its core, assuming it is correct to see
> one Solr core as the association of one schema & one conf.
>
> There have been a few threads about multiple indexes and/or
> multiple/reloading schemas.
> From what I gathered, one solution stems from the 'multiple webapp
> instances
> deployment' and implies 'extracting' the static instance (at least the
> SolrCore) & thus host multiple Solr cores in one webapp.
>
> Obviously, the operations (queries/add/delete doc) would need to carry
> which
> core they are targeting (one 'core' being set as the 'default' for
> compatibility purpose).
> What will be the other big hurdles, the ones that could even preclude the
> very idea ? (caches handling, updater threads, HA features...)
> Are there any easier routes (class-loaders, 'provisional' schema...) ?
>
> Any advice welcome. Thanks.
> Henri
>
>
> --
> View this message in context:
> http://www.nabble.com/Embedding-Solr-vs-Lucene%2C-multiple-Solr-cores--tf3572324.html#a9981355
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: Embedding Solr vs Lucene, multiple Solr cores?

[Ryan McKinley <ry...@gmail.com>]

On 4/13/07, Henrib <hb...@gmail.com> wrote:
>
> I'm trying to choose between embedding Lucene versus embedding Solr in one
> webapp.
>
> In Solr terms, functional requirements would more or less lead to multiple
> schema & conf (need CRUD/generation on those) and deployment constraints
> imply one webapp instance.

Do you really need multiple schema's?  multiple indexes?  This has
been posted many times (i thought i needed it too!) - it turns out
most cases can easily be taken care of by putting multiple document
types in the same index and including a "type" field.  You could have
a single schema with names for common fields, or one that has a prefix
for each type; either "title" or "typeA_title", "typeB_title" - the
common name approach can be easier because it makes it easier to
search across types.


> -Embed Solr but refactor 'some' of its core, assuming it is correct to see
> one Solr core as the association of one schema & one conf.

If you absolutely need multiple indexes, It will probably be easier to
fudge the single webapp requirement then to refactor solr to remove
the static singleton SolrCore.getSolrCore().

ryan