You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by bbarani <bb...@gmail.com> on 2013/04/23 19:23:56 UTC

Is there a way to load multiple schema when using zookeeper?

I have used multiple schema files by using multiple cores but not sure if I
will be able to use multiple schema configuration when integrating SOLR with
zookeeper. Can someone please let me know if its possible and if so, how?



--
View this message in context: http://lucene.472066.n3.nabble.com/Is-there-a-way-to-load-multiple-schema-when-using-zookeeper-tp4058358.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Is there a way to load multiple schema when using zookeeper?

Posted by Timothy Potter <th...@gmail.com>.
Yes - better use of existing resources. In this case, the chroot would
be helpful to keep Solr znodes separate from HBase. For the most part,
Solr in steady-state doesn't put a lot of stress on Zookeeper, for the
most part my zk nodes are snoozing.

On Tue, Apr 23, 2013 at 1:46 PM, Furkan KAMACI <fu...@gmail.com> wrote:
> If I have a Zookeper Cluster for my Hbase Cluster already, can I use same
> Zookeper cluster for my SolrCloud too?
>
> 2013/4/23 Timothy Potter <th...@gmail.com>
>
>> Ah cool, thanks for clarifying Chris - some of that multi-config
>> management stuff gets confusing but much clearer from your
>> description.
>>
>> Cheers,
>> Tim
>>
>> On Tue, Apr 23, 2013 at 11:36 AM, Chris Hostetter
>> <ho...@fucit.org> wrote:
>> >
>> > : Yes, you can effectively chroot all the configs for a collection (to
>> > : support multiple collections in same ensemble) - see wiki:
>> > : http://wiki.apache.org/solr/SolrCloud#Zookeeper_chroot
>> >
>> > I don't think chroot is suitable for what's being asked about here ...
>> > that would completely isolate two cloud clusters from eachother.
>> >
>> > i believe the intent of the question is asking about having a cloud
>> > cluster in which multiple collections exist, and some collections use
>> > differnet schema.xml files then other collections
>> >
>> > the short answer is: absolutely, each collecion can use completley
>> > differnet sets of configs (just like in non-cloud mode each core can use
>> > distinct configs) but to help make management easier when you have many
>> > collections re-using the same set of configs there is the concept of a
>> > "config set" ... you can push a set of configs into ZK with a specific
>> > "configName", nad then refer to that config set name when creating
>> > collections...
>> >
>> >
>> https://wiki.apache.org/solr/SolrCloud#Getting_your_Configuration_Files_into_ZooKeeper
>> >
>> http://wiki.apache.org/solr/SolrCloud#Managing_collections_via_the_Collections_API
>> >
>> >
>> > -Hoss
>>

Re: Is there a way to load multiple schema when using zookeeper?

Posted by sathish_ix <sk...@inautix.co.in>.
Hi,

we were able to accomplish this by single collection.

Zookeeper :

create separate node for each shards, and upload the dbconfig file under
shards.

eg : /config/config1/shard1
      /config/config1/shard2
      /config/config1/shard3

In the solrconfig.xml,

 <requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
                <lst name="defaults">
                        <str name="config">${dbconfig}</str>
                </lst>
        </requestHandler>


In solr.xml,

<?xml version="1.0" encoding="UTF-8" ?>
<solr persistent="true" zkHost="localhost:2181" >
  <cores defaultCoreName="core1" adminPath="/admin/cores"
zkClientTimeout="${zkClientTimeout:15000}"  host="${host:}" hostPort="9985"
hostContext="${hostContext:}">
       
          <core loadOnStartup="true" instanceDir="core1" transient="false"
name="core1" >
          <property name="dbconfig" value="shard1/db-data-config.xml" /> 
          </core>
 
  </cores>
</solr>

This way you can configure dbconfig file per shard.

Thanks,
Sathish 



--
View this message in context: http://lucene.472066.n3.nabble.com/Is-there-a-way-to-load-multiple-schema-when-using-zookeeper-tp4058358p4068820.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Is there a way to load multiple schema when using zookeeper?

Posted by sathish_ix <sk...@inautix.co.in>.
Hi,

we were able to accomplish this by single collection.

Zookeeper :

create separate node for each shards, and upload the dbconfig file under
shards.

eg : /config/config1/shard1
      /config/config1/shard2
      /config/config1/shard3

In the solrconfig.xml,

 <requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
                <lst name="defaults">
                        <str name="config">${dbconfig}</str>
                </lst>
        </requestHandler>


In solr.xml,

<?xml version="1.0" encoding="UTF-8" ?>
<solr persistent="true" zkHost="localhost:2181" >
  <cores defaultCoreName="core1" adminPath="/admin/cores"
zkClientTimeout="${zkClientTimeout:15000}"  host="${host:}" hostPort="9985"
hostContext="${hostContext:}">
       
          <core loadOnStartup="true" instanceDir="core1" transient="false"
name="core1" >
          <property name="dbconfig" value="shard1/db-data-config.xml" /> 
          </core>
 
  </cores>
</solr>

This way you can configure dbconfig file per shard.

Thanks,
Sathish 



--
View this message in context: http://lucene.472066.n3.nabble.com/Is-there-a-way-to-load-multiple-schema-when-using-zookeeper-tp4058358p4068821.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Is there a way to load multiple schema when using zookeeper?

Posted by Furkan KAMACI <fu...@gmail.com>.
I will use Nutch with map reduce to crawl huge data and use SolrCloud for
many users with high response time. Actually I wonder about performance
issues separating Zookeper cluster or using them for both Hbase and Solr.


2013/4/23 Shawn Heisey <so...@elyograg.org>

> On 4/23/2013 1:46 PM, Furkan KAMACI wrote:
>
>> If I have a Zookeper Cluster for my Hbase Cluster already, can I use same
>> Zookeper cluster for my SolrCloud too?
>>
>
> Yes, you can.  It is strongly recommended that you use a chroot with the
> zkHost parameter if you are sharing zookeeper.  It's a really good idea to
> use a chroot even if you're not sharing.  Here's an example zkHost
> parameter with a chroot of /mysolr.
>
> zoo1.example.com:2181,zoo2.example.com:2181,zoo3.example.com:2181/mysolr
>
> You only specify the chroot once at the end, not on every host entry. This
> information is also in the zookeeper documentation.
>
> Thanks,
> Shawn
>
>

Re: Is there a way to load multiple schema when using zookeeper?

Posted by Shawn Heisey <so...@elyograg.org>.
On 4/23/2013 1:46 PM, Furkan KAMACI wrote:
> If I have a Zookeper Cluster for my Hbase Cluster already, can I use same
> Zookeper cluster for my SolrCloud too?

Yes, you can.  It is strongly recommended that you use a chroot with the 
zkHost parameter if you are sharing zookeeper.  It's a really good idea 
to use a chroot even if you're not sharing.  Here's an example zkHost 
parameter with a chroot of /mysolr.

zoo1.example.com:2181,zoo2.example.com:2181,zoo3.example.com:2181/mysolr

You only specify the chroot once at the end, not on every host entry. 
This information is also in the zookeeper documentation.

Thanks,
Shawn

Re: Is there a way to load multiple schema when using zookeeper?

Posted by Furkan KAMACI <fu...@gmail.com>.
If I have a Zookeper Cluster for my Hbase Cluster already, can I use same
Zookeper cluster for my SolrCloud too?

2013/4/23 Timothy Potter <th...@gmail.com>

> Ah cool, thanks for clarifying Chris - some of that multi-config
> management stuff gets confusing but much clearer from your
> description.
>
> Cheers,
> Tim
>
> On Tue, Apr 23, 2013 at 11:36 AM, Chris Hostetter
> <ho...@fucit.org> wrote:
> >
> > : Yes, you can effectively chroot all the configs for a collection (to
> > : support multiple collections in same ensemble) - see wiki:
> > : http://wiki.apache.org/solr/SolrCloud#Zookeeper_chroot
> >
> > I don't think chroot is suitable for what's being asked about here ...
> > that would completely isolate two cloud clusters from eachother.
> >
> > i believe the intent of the question is asking about having a cloud
> > cluster in which multiple collections exist, and some collections use
> > differnet schema.xml files then other collections
> >
> > the short answer is: absolutely, each collecion can use completley
> > differnet sets of configs (just like in non-cloud mode each core can use
> > distinct configs) but to help make management easier when you have many
> > collections re-using the same set of configs there is the concept of a
> > "config set" ... you can push a set of configs into ZK with a specific
> > "configName", nad then refer to that config set name when creating
> > collections...
> >
> >
> https://wiki.apache.org/solr/SolrCloud#Getting_your_Configuration_Files_into_ZooKeeper
> >
> http://wiki.apache.org/solr/SolrCloud#Managing_collections_via_the_Collections_API
> >
> >
> > -Hoss
>

Re: Is there a way to load multiple schema when using zookeeper?

Posted by Timothy Potter <th...@gmail.com>.
Ah cool, thanks for clarifying Chris - some of that multi-config
management stuff gets confusing but much clearer from your
description.

Cheers,
Tim

On Tue, Apr 23, 2013 at 11:36 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : Yes, you can effectively chroot all the configs for a collection (to
> : support multiple collections in same ensemble) - see wiki:
> : http://wiki.apache.org/solr/SolrCloud#Zookeeper_chroot
>
> I don't think chroot is suitable for what's being asked about here ...
> that would completely isolate two cloud clusters from eachother.
>
> i believe the intent of the question is asking about having a cloud
> cluster in which multiple collections exist, and some collections use
> differnet schema.xml files then other collections
>
> the short answer is: absolutely, each collecion can use completley
> differnet sets of configs (just like in non-cloud mode each core can use
> distinct configs) but to help make management easier when you have many
> collections re-using the same set of configs there is the concept of a
> "config set" ... you can push a set of configs into ZK with a specific
> "configName", nad then refer to that config set name when creating
> collections...
>
> https://wiki.apache.org/solr/SolrCloud#Getting_your_Configuration_Files_into_ZooKeeper
> http://wiki.apache.org/solr/SolrCloud#Managing_collections_via_the_Collections_API
>
>
> -Hoss

Re: Is there a way to load multiple schema when using zookeeper?

Posted by Chris Hostetter <ho...@fucit.org>.
: Yes, you can effectively chroot all the configs for a collection (to
: support multiple collections in same ensemble) - see wiki:
: http://wiki.apache.org/solr/SolrCloud#Zookeeper_chroot

I don't think chroot is suitable for what's being asked about here ... 
that would completely isolate two cloud clusters from eachother.

i believe the intent of the question is asking about having a cloud 
cluster in which multiple collections exist, and some collections use 
differnet schema.xml files then other collections

the short answer is: absolutely, each collecion can use completley 
differnet sets of configs (just like in non-cloud mode each core can use 
distinct configs) but to help make management easier when you have many 
collections re-using the same set of configs there is the concept of a 
"config set" ... you can push a set of configs into ZK with a specific 
"configName", nad then refer to that config set name when creating 
collections... 

https://wiki.apache.org/solr/SolrCloud#Getting_your_Configuration_Files_into_ZooKeeper
http://wiki.apache.org/solr/SolrCloud#Managing_collections_via_the_Collections_API


-Hoss

Re: Is there a way to load multiple schema when using zookeeper?

Posted by Timothy Potter <th...@gmail.com>.
Yes, you can effectively chroot all the configs for a collection (to
support multiple collections in same ensemble) - see wiki:
http://wiki.apache.org/solr/SolrCloud#Zookeeper_chroot

On Tue, Apr 23, 2013 at 11:23 AM, bbarani <bb...@gmail.com> wrote:
>
> I have used multiple schema files by using multiple cores but not sure if I
> will be able to use multiple schema configuration when integrating SOLR with
> zookeeper. Can someone please let me know if its possible and if so, how?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Is-there-a-way-to-load-multiple-schema-when-using-zookeeper-tp4058358.html
> Sent from the Solr - User mailing list archive at Nabble.com.