You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Leo Prince <le...@gmail.com> on 2017/12/11 11:08:07 UTC
Specific cores/collections to specific nodes
Hi,
I just wanted to delegate specific Solr nodes to specific collections. As
per my understanding, when a collection creates in Solr7, core is
automatically creates in the backend. Since my cores are getting different
volume of traffic, I wanted to delegate specific collections to specific
nodes. One core is read intensive with less write and other is write
intensive with less read so dont want to mix their IO together. Earlier in
Solr4, we were able to hard code it in tomcat solr.xml, similarly is there
any ways that we can do this in solr7..?
Thanks in advance.
Leo Prince.
Re: Specific cores/collections to specific nodes
Posted by Erick Erickson <er...@gmail.com>.
Glad that worked. No, there's no way to hard-code as the old solr.xml,
that's pretty antithetical to SolrCloud.
IIRC, though, you're making it too complicated ;)....
Machine N goes down that hosts collection_shard1_replica32
Bring machine N+1 up with Solr
Collections API:
ADDREPLCIA with the "node" parmeter for collection_shard1. That puts
the new replica right where you want it and it gets synchronized with
the leader and starts being used for search.
DELETEREPLICA for collection_shard1_replica32
Best,
Erick
On Tue, Dec 12, 2017 at 9:47 PM, Leo Prince
<le...@gmail.com> wrote:
> Thanks Erick.
>
> createNodeSet works perfect. A nice way of allocating collections among
> nodes. When I went through the docs, host tag just noticed so thought to
> make use of it, however createNodeSet works just fine for me; and
> createNodeSet.shuffle too.
>
> Just wanted to know one more thing. Is there any way to hard code the
> collections with each node like we were able to hardcode the core details
> in tomcat solr.xml back in Solr4. I just asked this because, Suppose if one
> node went down like irrecoverable, I have to get one machine back to the
> cluster to replace it and maintain 2n+1. If we were able to hardcode it
> within the instance, all I have to do is, launch an instance with it's
> concerned image with the concerned static IP configured in ZK. Any advise
> is appreciated.
>
> Leo Prince
>
> On Tue, Dec 12, 2017 at 9:42 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> Wait, rule-based placement isn't what I was talking about at all. You
>> don't need to knock yourself out understanding the rule based replica
>> placement, which is being replaced by "Policies" in 7x anyway.
>>
>> You can just use createNodeSet in the collection CREATE command. It's
>> a list of nodes (as in the live_nodes znode). Optionally you can also
>> disable "shuffling" with reateNodeSet.shuffle, so the shards are
>> assigned to the createNodeSet in order. Or you can use the EMPTY
>> keyword that allows you to add replicas via ADDREPLICA one by one on
>> specific nodes (again from the live_nodes znode).
>>
>> Why do you want to use rules instead?
>>
>> Best,
>> Erick
>>
>> On Tue, Dec 12, 2017 at 12:35 AM, Leo Prince
>> <le...@gmail.com> wrote:
>> > Thanks Erick. I went through the doc
>> > https://lucidworks.com/2015/05/12/rule-based-replica-
>> assignment-solrcloud/
>> > and it helped on how to use "host" tag. However can I use two values to
>> > same key.?
>> >
>> > I can do host:!1.1.1.1 but can I do something like host:!1.1.1.1,!2.2.2.2
>> > or host:!1.1.1.1,host:!2.2.2.2
>> >
>> > I dont see it in the doc
>> >
>> > Thanks
>> >
>> > On Mon, Dec 11, 2017 at 9:46 PM, Erick Erickson <erickerickson@gmail.com
>> >
>> > wrote:
>> >
>> >> Take a look at the collections API CREATE command, especially the
>> >> "createNodeSet". One variant lets you specify the nodes used to
>> >> distribute the collection with limited control over what core goes
>> >> where through the createNodeSet.shuffle parameter.
>> >>
>> >> Alternatively you can use "EMPTY" for the createNodeSet and Solr will
>> >> create the skeleton of the collection in ZK but not create any actual
>> >> replicas. You can then use ADDREPLICA to place each and every node
>> >> exactly where you want it with the "node" parameter of ADDREPLCIA.
>> >>
>> >> In both cases, the "node" is the string you see under the "live_nodes"
>> >> znode.
>> >>
>> >> Best,
>> >> Erick
>> >>
>> >> On Mon, Dec 11, 2017 at 3:08 AM, Leo Prince
>> >> <le...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I just wanted to delegate specific Solr nodes to specific
>> collections. As
>> >> > per my understanding, when a collection creates in Solr7, core is
>> >> > automatically creates in the backend. Since my cores are getting
>> >> different
>> >> > volume of traffic, I wanted to delegate specific collections to
>> specific
>> >> > nodes. One core is read intensive with less write and other is write
>> >> > intensive with less read so dont want to mix their IO together.
>> Earlier
>> >> in
>> >> > Solr4, we were able to hard code it in tomcat solr.xml, similarly is
>> >> there
>> >> > any ways that we can do this in solr7..?
>> >> >
>> >> > Thanks in advance.
>> >> > Leo Prince.
>> >>
>>
Re: Specific cores/collections to specific nodes
Posted by Leo Prince <le...@gmail.com>.
Thanks Erick.
createNodeSet works perfect. A nice way of allocating collections among
nodes. When I went through the docs, host tag just noticed so thought to
make use of it, however createNodeSet works just fine for me; and
createNodeSet.shuffle too.
Just wanted to know one more thing. Is there any way to hard code the
collections with each node like we were able to hardcode the core details
in tomcat solr.xml back in Solr4. I just asked this because, Suppose if one
node went down like irrecoverable, I have to get one machine back to the
cluster to replace it and maintain 2n+1. If we were able to hardcode it
within the instance, all I have to do is, launch an instance with it's
concerned image with the concerned static IP configured in ZK. Any advise
is appreciated.
Leo Prince
On Tue, Dec 12, 2017 at 9:42 PM, Erick Erickson <er...@gmail.com>
wrote:
> Wait, rule-based placement isn't what I was talking about at all. You
> don't need to knock yourself out understanding the rule based replica
> placement, which is being replaced by "Policies" in 7x anyway.
>
> You can just use createNodeSet in the collection CREATE command. It's
> a list of nodes (as in the live_nodes znode). Optionally you can also
> disable "shuffling" with reateNodeSet.shuffle, so the shards are
> assigned to the createNodeSet in order. Or you can use the EMPTY
> keyword that allows you to add replicas via ADDREPLICA one by one on
> specific nodes (again from the live_nodes znode).
>
> Why do you want to use rules instead?
>
> Best,
> Erick
>
> On Tue, Dec 12, 2017 at 12:35 AM, Leo Prince
> <le...@gmail.com> wrote:
> > Thanks Erick. I went through the doc
> > https://lucidworks.com/2015/05/12/rule-based-replica-
> assignment-solrcloud/
> > and it helped on how to use "host" tag. However can I use two values to
> > same key.?
> >
> > I can do host:!1.1.1.1 but can I do something like host:!1.1.1.1,!2.2.2.2
> > or host:!1.1.1.1,host:!2.2.2.2
> >
> > I dont see it in the doc
> >
> > Thanks
> >
> > On Mon, Dec 11, 2017 at 9:46 PM, Erick Erickson <erickerickson@gmail.com
> >
> > wrote:
> >
> >> Take a look at the collections API CREATE command, especially the
> >> "createNodeSet". One variant lets you specify the nodes used to
> >> distribute the collection with limited control over what core goes
> >> where through the createNodeSet.shuffle parameter.
> >>
> >> Alternatively you can use "EMPTY" for the createNodeSet and Solr will
> >> create the skeleton of the collection in ZK but not create any actual
> >> replicas. You can then use ADDREPLICA to place each and every node
> >> exactly where you want it with the "node" parameter of ADDREPLCIA.
> >>
> >> In both cases, the "node" is the string you see under the "live_nodes"
> >> znode.
> >>
> >> Best,
> >> Erick
> >>
> >> On Mon, Dec 11, 2017 at 3:08 AM, Leo Prince
> >> <le...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I just wanted to delegate specific Solr nodes to specific
> collections. As
> >> > per my understanding, when a collection creates in Solr7, core is
> >> > automatically creates in the backend. Since my cores are getting
> >> different
> >> > volume of traffic, I wanted to delegate specific collections to
> specific
> >> > nodes. One core is read intensive with less write and other is write
> >> > intensive with less read so dont want to mix their IO together.
> Earlier
> >> in
> >> > Solr4, we were able to hard code it in tomcat solr.xml, similarly is
> >> there
> >> > any ways that we can do this in solr7..?
> >> >
> >> > Thanks in advance.
> >> > Leo Prince.
> >>
>
Re: Specific cores/collections to specific nodes
Posted by Erick Erickson <er...@gmail.com>.
Wait, rule-based placement isn't what I was talking about at all. You
don't need to knock yourself out understanding the rule based replica
placement, which is being replaced by "Policies" in 7x anyway.
You can just use createNodeSet in the collection CREATE command. It's
a list of nodes (as in the live_nodes znode). Optionally you can also
disable "shuffling" with reateNodeSet.shuffle, so the shards are
assigned to the createNodeSet in order. Or you can use the EMPTY
keyword that allows you to add replicas via ADDREPLICA one by one on
specific nodes (again from the live_nodes znode).
Why do you want to use rules instead?
Best,
Erick
On Tue, Dec 12, 2017 at 12:35 AM, Leo Prince
<le...@gmail.com> wrote:
> Thanks Erick. I went through the doc
> https://lucidworks.com/2015/05/12/rule-based-replica-assignment-solrcloud/
> and it helped on how to use "host" tag. However can I use two values to
> same key.?
>
> I can do host:!1.1.1.1 but can I do something like host:!1.1.1.1,!2.2.2.2
> or host:!1.1.1.1,host:!2.2.2.2
>
> I dont see it in the doc
>
> Thanks
>
> On Mon, Dec 11, 2017 at 9:46 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> Take a look at the collections API CREATE command, especially the
>> "createNodeSet". One variant lets you specify the nodes used to
>> distribute the collection with limited control over what core goes
>> where through the createNodeSet.shuffle parameter.
>>
>> Alternatively you can use "EMPTY" for the createNodeSet and Solr will
>> create the skeleton of the collection in ZK but not create any actual
>> replicas. You can then use ADDREPLICA to place each and every node
>> exactly where you want it with the "node" parameter of ADDREPLCIA.
>>
>> In both cases, the "node" is the string you see under the "live_nodes"
>> znode.
>>
>> Best,
>> Erick
>>
>> On Mon, Dec 11, 2017 at 3:08 AM, Leo Prince
>> <le...@gmail.com> wrote:
>> > Hi,
>> >
>> > I just wanted to delegate specific Solr nodes to specific collections. As
>> > per my understanding, when a collection creates in Solr7, core is
>> > automatically creates in the backend. Since my cores are getting
>> different
>> > volume of traffic, I wanted to delegate specific collections to specific
>> > nodes. One core is read intensive with less write and other is write
>> > intensive with less read so dont want to mix their IO together. Earlier
>> in
>> > Solr4, we were able to hard code it in tomcat solr.xml, similarly is
>> there
>> > any ways that we can do this in solr7..?
>> >
>> > Thanks in advance.
>> > Leo Prince.
>>
Re: Specific cores/collections to specific nodes
Posted by Leo Prince <le...@gmail.com>.
Thanks Erick. I went through the doc
https://lucidworks.com/2015/05/12/rule-based-replica-assignment-solrcloud/
and it helped on how to use "host" tag. However can I use two values to
same key.?
I can do host:!1.1.1.1 but can I do something like host:!1.1.1.1,!2.2.2.2
or host:!1.1.1.1,host:!2.2.2.2
I dont see it in the doc
Thanks
On Mon, Dec 11, 2017 at 9:46 PM, Erick Erickson <er...@gmail.com>
wrote:
> Take a look at the collections API CREATE command, especially the
> "createNodeSet". One variant lets you specify the nodes used to
> distribute the collection with limited control over what core goes
> where through the createNodeSet.shuffle parameter.
>
> Alternatively you can use "EMPTY" for the createNodeSet and Solr will
> create the skeleton of the collection in ZK but not create any actual
> replicas. You can then use ADDREPLICA to place each and every node
> exactly where you want it with the "node" parameter of ADDREPLCIA.
>
> In both cases, the "node" is the string you see under the "live_nodes"
> znode.
>
> Best,
> Erick
>
> On Mon, Dec 11, 2017 at 3:08 AM, Leo Prince
> <le...@gmail.com> wrote:
> > Hi,
> >
> > I just wanted to delegate specific Solr nodes to specific collections. As
> > per my understanding, when a collection creates in Solr7, core is
> > automatically creates in the backend. Since my cores are getting
> different
> > volume of traffic, I wanted to delegate specific collections to specific
> > nodes. One core is read intensive with less write and other is write
> > intensive with less read so dont want to mix their IO together. Earlier
> in
> > Solr4, we were able to hard code it in tomcat solr.xml, similarly is
> there
> > any ways that we can do this in solr7..?
> >
> > Thanks in advance.
> > Leo Prince.
>
Re: Specific cores/collections to specific nodes
Posted by Erick Erickson <er...@gmail.com>.
Take a look at the collections API CREATE command, especially the
"createNodeSet". One variant lets you specify the nodes used to
distribute the collection with limited control over what core goes
where through the createNodeSet.shuffle parameter.
Alternatively you can use "EMPTY" for the createNodeSet and Solr will
create the skeleton of the collection in ZK but not create any actual
replicas. You can then use ADDREPLICA to place each and every node
exactly where you want it with the "node" parameter of ADDREPLCIA.
In both cases, the "node" is the string you see under the "live_nodes" znode.
Best,
Erick
On Mon, Dec 11, 2017 at 3:08 AM, Leo Prince
<le...@gmail.com> wrote:
> Hi,
>
> I just wanted to delegate specific Solr nodes to specific collections. As
> per my understanding, when a collection creates in Solr7, core is
> automatically creates in the backend. Since my cores are getting different
> volume of traffic, I wanted to delegate specific collections to specific
> nodes. One core is read intensive with less write and other is write
> intensive with less read so dont want to mix their IO together. Earlier in
> Solr4, we were able to hard code it in tomcat solr.xml, similarly is there
> any ways that we can do this in solr7..?
>
> Thanks in advance.
> Leo Prince.