Sharing SolrCloud collection configs w/overrides

[Tim Vaillancourt <ti...@elementspace.com>]

Hey guys,

I have a situation where I have a lot of collections that share the same
core config in Zookeeper. For each of my SolrCloud collections, 99.9% of
the config (schema.xml, solrcloud.xml) are the same, only the
DataImportHandler parameters are different for different database
names/credentials, per collection.

To provide the different DIH credentials per collection, I currently upload
many copies of the exact-same Solr config dir with 1 Xincluded file with
the 4-5 database parameters that are different alongside the schema.xml and
solrconfig.xml.

I don't feel this ideal and is wasting space in Zookeeper considering most
of my configs are duplicated.

At a high level, is there a way for me to share one config in Zookeeper
while having minor overrides to the variables?

Is there a way for me to XInclude a file outside of my Zookeeper config
dir, ie: could I XInclude arbitrary locations in Zookeeper so that I can
have the same config dir for all collections and a file in Zookeeper that
is external to the common config dir to apply the collection-specific
overrides?

To extend my question for Solr 4.4 core.properties files: am I stuck in the
same boat under Solr 4.4 if I have say 10 collections sharing one config,
but I want each to have a unique core.properties?

Cheers!

Tim

Re: Sharing SolrCloud collection configs w/overrides

[Tim Vaillancourt <ti...@elementspace.com>]

Here you go Erick, feel free to update this.

I am unable to assign to you, but asked for someone to do so:

https://issues.apache.org/jira/browse/SOLR-5208

Cheers,

Tim

On 21/08/13 10:40 AM, Tim Vaillancourt wrote:
> Well, the mention of DIH is a bit off-topic. I'll simplify and say all 
> I need is the ability to set ANY variables in solrconfig.xml without 
> having to make N number of copies of the same configuration to achieve 
> that. Essentially I need 10+ collections to use the exact same config 
> dir in Zookeeper with minor/trivial differences set in variables.
>
> Your proposal of taking in values at core creation-time is a neat one 
> and would be a very flexible solution for a lot of use cases. My only 
> concern for my really-specific use cae is that I'd be setting DB 
> user/passwords via plain-text HTTP calls, but having this feature is 
> better than not.
>
> In a perfect world I'd like to be able to include files in Zookeeper 
> (like XInclude) that are outside the common config dir (eg: 
> '/configs/sharedconfig') all the collections would be sharing. On the 
> other hand, that sort of solution would open up the Zookeeper layout 
> to arbitrary files and could end up in a nightmare if not done 
> carefully, however.
>
> Would it be possible for Solr to support specifying multiple configs 
> at collection creation, that are merged or concatenated. This idea 
> sounds terrible to me even at this moment, but I wonder if there is 
> something in there..
>
> Tim

Re: Sharing SolrCloud collection configs w/overrides

[Tim Vaillancourt <ti...@elementspace.com>]

Well, the mention of DIH is a bit off-topic. I'll simplify and say all I
need is the ability to set ANY variables in solrconfig.xml without having
to make N number of copies of the same configuration to achieve that.
Essentially I need 10+ collections to use the exact same config dir in
Zookeeper with minor/trivial differences set in variables.

Your proposal of taking in values at core creation-time is a neat one and
would be a very flexible solution for a lot of use cases. My only concern
for my really-specific use cae is that I'd be setting DB user/passwords via
plain-text HTTP calls, but having this feature is better than not.

In a perfect world I'd like to be able to include files in Zookeeper (like
XInclude) that are outside the common config dir (eg:
'/configs/sharedconfig') all the collections would be sharing. On the other
hand, that sort of solution would open up the Zookeeper layout to arbitrary
files and could end up in a nightmare if not done carefully, however.

Would it be possible for Solr to support specifying multiple configs at
collection creation, that are merged or concatenated. This idea sounds
terrible to me even at this moment, but I wonder if there is something in
there..

Tim

Re: Sharing SolrCloud collection configs w/overrides

[Erick Erickson <er...@gmail.com>]

Hmmm. I'm going to leave the DIH stuff for someone else, but could
you raise a JIRA (and assign it to me) to think about a way to add
a core.properties file to the collection creation step?

I haven't thought it through very well, but currently I think we just assign
some defaults. Some thought that needs to be put into what
kinds of things we'll allow to be set, I think you could shoot yourself in
the foot.

Maybe it's as simple as allowing more params for creation like
collection.coreName where each param of the form collection.blah=blort
gets an entry in the properties file blah=blort? Would that work for your
case?

Best,
Erick


On Tue, Aug 20, 2013 at 2:22 PM, Tim Vaillancourt <ti...@elementspace.com>wrote:

> Hey guys,
>
> I have a situation where I have a lot of collections that share the same
> core config in Zookeeper. For each of my SolrCloud collections, 99.9% of
> the config (schema.xml, solrcloud.xml) are the same, only the
> DataImportHandler parameters are different for different database
> names/credentials, per collection.
>
> To provide the different DIH credentials per collection, I currently upload
> many copies of the exact-same Solr config dir with 1 Xincluded file with
> the 4-5 database parameters that are different alongside the schema.xml and
> solrconfig.xml.
>
> I don't feel this ideal and is wasting space in Zookeeper considering most
> of my configs are duplicated.
>
> At a high level, is there a way for me to share one config in Zookeeper
> while having minor overrides to the variables?
>
> Is there a way for me to XInclude a file outside of my Zookeeper config
> dir, ie: could I XInclude arbitrary locations in Zookeeper so that I can
> have the same config dir for all collections and a file in Zookeeper that
> is external to the common config dir to apply the collection-specific
> overrides?
>
> To extend my question for Solr 4.4 core.properties files: am I stuck in the
> same boat under Solr 4.4 if I have say 10 collections sharing one config,
> but I want each to have a unique core.properties?
>
> Cheers!
>
> Tim
>