You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by David Santamauro <da...@gmail.com> on 2013/12/22 12:48:25 UTC
adding a node to SolrCloud
Hi,
I have an 8-node setup currently with 1 shard per node (no redundancy).
These 8 nodes are smaller machines not capable of supporting the entire
collection..
I have another machine resource that can act as other node and this last
node is capable of holding the entire collection. I'd like to make this
large node hold a replica of all the other 8 nodes' shards for
redundancy as well as performance.
Is this possible? ... and what would be the configuration magic needed
to accomplish this? This collection is very large (1.4TB) and as you can
imagine, I don't want to fiddle around trying configuration alternatives
assuming each try would lead to a replication of 1.4TB of data.
Any help would be greatly appreciated.
David
Re: adding a node to SolrCloud
Posted by David Santamauro <da...@gmail.com>.
On 12/26/2013 02:29 PM, Shawn Heisey wrote:
> On 12/24/2013 8:35 AM, David Santamauro wrote:
>>> You may have one or more of the SolrCloud 'bootstrap' options on the
>>> startup commandline. The bootstrap options are intended to be used
>>> once, in order to bootstrap from a non-SolrCloud setup to a SolrCloud
>>> setup.
>>
>> No, no unnecessary options. I manually bootstrapped a common config.
>
> I have no idea what might be wrong here.
>
>>> Between the Collections API and the CoreAdmin API, you should never need
>>> to edit solr.xml (if using the pre-4.4 format) or core.properties files
>>> (if using core discovery, available 4.4 and later) directly.
>>
>> Now this I don't understand. If I have created cores through the
>> CoreAdmin API, how is solr.xml affected? If I don't edit it, how does
>> SOLR know what cores it has to expose to a distributed collection?
>
> If you are using the old-style solr.xml (which will be supported through
> all future 4.x versions, but not 5.0), then core definitions are stored
> in solr.xml and the contents of the file are changed by many of the
> CoreAdmin API actions. The Collections API calls the CoreAdmin API on
> servers throughout the cloud.
I have never experienced tomcat or the SOLR webapp create, modify or
otherwise touch in anyway the solr.xml file. I have always had to add
the necessary core definition manually.
> http://wiki.apache.org/solr/Solr.xml%20%28supported%20through%204.x%29
>
> If you are using the core discovery format, which was made available in
> working form in version 4.4, then solr.xml does NOT contain core
> definitions. The main example in 4.4 and later uses the new format.
> Cores are discovered at Solr startup by crawling the filesystem from a
> root starting point looking for core.properties files. In this mode,
> solr.xml is fairly static.
>
> http://wiki.apache.org/solr/Solr.xml%204.4%20and%20beyond
> http://wiki.apache.org/solr/Core%20Discovery%20%284.4%20and%20beyond%29
I'll begin exploring this new format, thanks for the help and links.
David
Re: adding a node to SolrCloud
Posted by Shawn Heisey <so...@elyograg.org>.
On 12/24/2013 8:35 AM, David Santamauro wrote:
>> You may have one or more of the SolrCloud 'bootstrap' options on the
>> startup commandline. The bootstrap options are intended to be used
>> once, in order to bootstrap from a non-SolrCloud setup to a SolrCloud
>> setup.
>
> No, no unnecessary options. I manually bootstrapped a common config.
I have no idea what might be wrong here.
>> Between the Collections API and the CoreAdmin API, you should never need
>> to edit solr.xml (if using the pre-4.4 format) or core.properties files
>> (if using core discovery, available 4.4 and later) directly.
>
> Now this I don't understand. If I have created cores through the
> CoreAdmin API, how is solr.xml affected? If I don't edit it, how does
> SOLR know what cores it has to expose to a distributed collection?
If you are using the old-style solr.xml (which will be supported through
all future 4.x versions, but not 5.0), then core definitions are stored
in solr.xml and the contents of the file are changed by many of the
CoreAdmin API actions. The Collections API calls the CoreAdmin API on
servers throughout the cloud.
http://wiki.apache.org/solr/Solr.xml%20%28supported%20through%204.x%29
If you are using the core discovery format, which was made available in
working form in version 4.4, then solr.xml does NOT contain core
definitions. The main example in 4.4 and later uses the new format.
Cores are discovered at Solr startup by crawling the filesystem from a
root starting point looking for core.properties files. In this mode,
solr.xml is fairly static.
http://wiki.apache.org/solr/Solr.xml%204.4%20and%20beyond
http://wiki.apache.org/solr/Core%20Discovery%20%284.4%20and%20beyond%29
Thanks,
Shawn
Re: adding a node to SolrCloud
Posted by David Santamauro <da...@gmail.com>.
On 12/23/2013 08:42 PM, Shawn Heisey wrote:
> On 12/23/2013 12:23 PM, David Santamauro wrote:
>> I managed to create 8 new cores and the Solr Admin cloud page showed
>> them wonderfully as active replicas.
>>
>> The only issue I have is what goes into solr.xml (I'm using tomcat)?
>>
>> Putting
>> <core name="..." />
>>
>> for each of the new cores I created seemed like the reasonable approach
>> but when I tested a tomcat restart, the distribution was messed up ...
>> for one thing, the cores on the new machine showed up as collections!
>> And tomcat never even made it to accept connections for some reason.
>>
>> I cleaned everything up with zookepper so my graph looks like it should
>> and I removed that new machine from the distribution (by removing zk
>> attributes) and restarted .. all is well again.
>>
>> Any idea what could have went wrong on tomcat restart?
>
> You may have one or more of the SolrCloud 'bootstrap' options on the
> startup commandline. The bootstrap options are intended to be used
> once, in order to bootstrap from a non-SolrCloud setup to a SolrCloud setup.
No, no unnecessary options. I manually bootstrapped a common config.
> Between the Collections API and the CoreAdmin API, you should never need
> to edit solr.xml (if using the pre-4.4 format) or core.properties files
> (if using core discovery, available 4.4 and later) directly.
Now this I don't understand. If I have created cores through the
CoreAdmin API, how is solr.xml affected? If I don't edit it, how does
SOLR know what cores it has to expose to a distributed collection?
thanks
David
Re: adding a node to SolrCloud
Posted by Shawn Heisey <so...@elyograg.org>.
On 12/23/2013 12:23 PM, David Santamauro wrote:
> I managed to create 8 new cores and the Solr Admin cloud page showed
> them wonderfully as active replicas.
>
> The only issue I have is what goes into solr.xml (I'm using tomcat)?
>
> Putting
> <core name="..." />
>
> for each of the new cores I created seemed like the reasonable approach
> but when I tested a tomcat restart, the distribution was messed up ...
> for one thing, the cores on the new machine showed up as collections!
> And tomcat never even made it to accept connections for some reason.
>
> I cleaned everything up with zookepper so my graph looks like it should
> and I removed that new machine from the distribution (by removing zk
> attributes) and restarted .. all is well again.
>
> Any idea what could have went wrong on tomcat restart?
You may have one or more of the SolrCloud 'bootstrap' options on the
startup commandline. The bootstrap options are intended to be used
once, in order to bootstrap from a non-SolrCloud setup to a SolrCloud setup.
Between the Collections API and the CoreAdmin API, you should never need
to edit solr.xml (if using the pre-4.4 format) or core.properties files
(if using core discovery, available 4.4 and later) directly.
Thanks,
Shawn
Re: adding a node to SolrCloud
Posted by David Santamauro <da...@gmail.com>.
Shawn,
I managed to create 8 new cores and the Solr Admin cloud page showed
them wonderfully as active replicas.
The only issue I have is what goes into solr.xml (I'm using tomcat)?
Putting
<core name="..." />
for each of the new cores I created seemed like the reasonable approach
but when I tested a tomcat restart, the distribution was messed up ...
for one thing, the cores on the new machine showed up as collections!
And tomcat never even made it to accept connections for some reason.
I cleaned everything up with zookepper so my graph looks like it should
and I removed that new machine from the distribution (by removing zk
attributes) and restarted .. all is well again.
Any idea what could have went wrong on tomcat restart?
thanks.
On 12/22/2013 09:48 PM, Shawn Heisey wrote:
> On 12/22/2013 2:10 PM, David Santamauro wrote:
>> My goal is to have a redundant copy of all 8 currently running, but
>> non-redundant shards. This setup (8 nodes with no replicas) was a test
>> and it has proven quite functional from a performance perspective.
>> Loading, though, takes almost 3 weeks so I'm really not in a position to
>> redesign the distribution, though I can add nodes.
>>
>> I have acquired another resource, a very large machine that I'd like to
>> use to hold the replicas of the currently deployed 8-nodes.
>>
>> I realize I can run 8 jetty/tomcats and accomplish my goal but that is a
>> maintenance headache and is really a last resort. I really would just
>> like to be able to deploy this big machine with 'numShards=8'.
>>
>> Is that possible or do I really need to have 8 other nodes running?
>
> You don't want to run more than one container or Solr instance per
> machine. Things can get very confused, and it's too much overhead.
> Also, you shouldn't start Solr with the numShards parameter on the
> commandline. That should be given when you create each collection.
>
> With existing collections, you can simply run the CoreAdmin CREATE
> action on the new node with more resources.
>
> http://wiki.apache.org/solr/SolrCloud#Creating_cores_via_CoreAdmin
>
> So you'd do something like this, once for each of the 8 existing parts:
>
> http://newnode:port/solr/admin/cores?action=CREATE&name=collname_shard1_replica2&collection=collname&shard=shard1
>
> It will automatically replicate the shard from its current leader.
>
> One thing to be aware of: With 1.4TB of index data, it might be
> impossible to keep enough of the index in RAM for good performance,
> unless the machine has a terabyte or more of RAM.
>
> http://wiki.apache.org/solr/SolrPerformanceProblems#OS_Disk_Cache
>
> Thanks,
> Shawn
>
Re: adding a node to SolrCloud
Posted by David Santamauro <da...@gmail.com>.
On 12/23/2013 05:43 PM, Greg Preston wrote:
> I believe you can just define multiple cores:
>
> <core default="true" instanceDir="shard1/"
> name="collectionName_shard1" shard="shard1"/>
> <core default="true" instanceDir="shard2/"
> name="collectionName_shard2" shard="shard2"/>
> ...
>
> (this is the old style solr.xml. I don't know how to do it in the newer style)
Yes, that is exactly what I did but somehow, the link between shards and
collections gets lost and everything gets very confused.
I guess I should have read more carefully about the valid parameters on
the <core> element. My problem was a missing attribute:
@collection="collection-name"
So the complete core definition that survives tomcat restarts:
<core name="core_shard_1"
collection="collection-name"
instanceDir="/solr/instance/dir"
config="solrconfig-standard.xml"
dataDir="/solr/data/dir/core_shard_1"
/>
David
Re: adding a node to SolrCloud
Posted by Greg Preston <gp...@marinsoftware.com>.
I believe you can just define multiple cores:
<core default="true" instanceDir="shard1/"
name="collectionName_shard1" shard="shard1"/>
<core default="true" instanceDir="shard2/"
name="collectionName_shard2" shard="shard2"/>
...
(this is the old style solr.xml. I don't know how to do it in the newer style)
Also, make sure you don't define a non-relative <dataDir> in
solrconfig.xml, or you may run into issues with cores trying to use
the same data dir.
-Greg
On Mon, Dec 23, 2013 at 2:16 PM, David Santamauro
<da...@gmail.com> wrote:
> On 12/23/2013 05:03 PM, Greg Preston wrote:
>>>
>>> Yes, I'm well aware of the performance implications, many of which are
>>> mitigated by 2TB of SSD and 512GB RAM
>>
>>
>> I've got a very similar setup in production. 2TB SSD, 256G RAM (128G
>> heaps), and 1 - 1.5 TB of index per node. We're in the process of
>> splitting that to multiple JVMs per host. GC pauses were causing ZK
>> timeouts (you should up that in solr.xml). And resync's after the
>> timeouts took long enough that a large tlog built up (we have near
>> continuous indexing), and we couldn't replay the tlog fast enough to
>> catch up to current.
>
>
> GC pauses are a huge issue in our current production environment (monolithic
> index) and general performance was meager, hence the move to a distributed
> design. We will have 8 nodes with ~ 200GB per node, one shard each and
> performance for single and most multi-term queries has become sub-second and
> throughput has increased 10-fold. Larger boolean queries can still take 2-3s
> but we can live with that.
>
> At any rate, I still can't figure out what my solr.xml is supposed to look
> like on the node with all 8 redundant shards.
>
> David
>
>
>
>> On Mon, Dec 23, 2013 at 2:31 AM, David Santamauro
>> <da...@gmail.com> wrote:
>>>
>>> On 12/22/2013 09:48 PM, Shawn Heisey wrote:
>>>>
>>>>
>>>> On 12/22/2013 2:10 PM, David Santamauro wrote:
>>>>>
>>>>>
>>>>> My goal is to have a redundant copy of all 8 currently running, but
>>>>> non-redundant shards. This setup (8 nodes with no replicas) was a test
>>>>> and it has proven quite functional from a performance perspective.
>>>>> Loading, though, takes almost 3 weeks so I'm really not in a position
>>>>> to
>>>>> redesign the distribution, though I can add nodes.
>>>>>
>>>>> I have acquired another resource, a very large machine that I'd like to
>>>>> use to hold the replicas of the currently deployed 8-nodes.
>>>>>
>>>>> I realize I can run 8 jetty/tomcats and accomplish my goal but that is
>>>>> a
>>>>> maintenance headache and is really a last resort. I really would just
>>>>> like to be able to deploy this big machine with 'numShards=8'.
>>>>>
>>>>> Is that possible or do I really need to have 8 other nodes running?
>>>>
>>>>
>>>>
>>>> You don't want to run more than one container or Solr instance per
>>>> machine. Things can get very confused, and it's too much overhead.
>>>
>>>
>>>>
>>>>
>>>> With existing collections, you can simply run the CoreAdmin CREATE
>>>>
>>>> action on the new node with more resources.
>>>>
>>>> So you'd do something like this, once for each of the 8 existing parts:
>>>>
>>>>
>>>>
>>>> http://newnode:port/solr/admin/cores?action=CREATE&name=collname_shard1_replica2&collection=collname&shard=shard1
>>>>
>>>> It will automatically replicate the shard from its current leader.
>>>
>>>
>>>
>>> Fantastic! Clearly my understanding of "collection", vs "core" vs "shard"
>>> was lacking but now I see the relationship better.
>>>
>>>
>>>>
>>>> One thing to be aware of: With 1.4TB of index data, it might be
>>>> impossible to keep enough of the index in RAM for good performance,
>>>> unless the machine has a terabyte or more of RAM.
>>>
>>>
>>>
>>> Yes, I'm well aware of the performance implications, many of which are
>>> mitigated by 2TB of SSD and 512GB RAM.
>>>
>>> Thanks for the nudge in the right direction. The first node/shard1 is
>>> replicating right now.
>>>
>>> David
>>>
>>>
>>>
>
Re: adding a node to SolrCloud
Posted by David Santamauro <da...@gmail.com>.
On 12/23/2013 05:03 PM, Greg Preston wrote:
>> Yes, I'm well aware of the performance implications, many of which are mitigated by 2TB of SSD and 512GB RAM
>
> I've got a very similar setup in production. 2TB SSD, 256G RAM (128G
> heaps), and 1 - 1.5 TB of index per node. We're in the process of
> splitting that to multiple JVMs per host. GC pauses were causing ZK
> timeouts (you should up that in solr.xml). And resync's after the
> timeouts took long enough that a large tlog built up (we have near
> continuous indexing), and we couldn't replay the tlog fast enough to
> catch up to current.
GC pauses are a huge issue in our current production environment
(monolithic index) and general performance was meager, hence the move to
a distributed design. We will have 8 nodes with ~ 200GB per node, one
shard each and performance for single and most multi-term queries has
become sub-second and throughput has increased 10-fold. Larger boolean
queries can still take 2-3s but we can live with that.
At any rate, I still can't figure out what my solr.xml is supposed to
look like on the node with all 8 redundant shards.
David
> On Mon, Dec 23, 2013 at 2:31 AM, David Santamauro
> <da...@gmail.com> wrote:
>> On 12/22/2013 09:48 PM, Shawn Heisey wrote:
>>>
>>> On 12/22/2013 2:10 PM, David Santamauro wrote:
>>>>
>>>> My goal is to have a redundant copy of all 8 currently running, but
>>>> non-redundant shards. This setup (8 nodes with no replicas) was a test
>>>> and it has proven quite functional from a performance perspective.
>>>> Loading, though, takes almost 3 weeks so I'm really not in a position to
>>>> redesign the distribution, though I can add nodes.
>>>>
>>>> I have acquired another resource, a very large machine that I'd like to
>>>> use to hold the replicas of the currently deployed 8-nodes.
>>>>
>>>> I realize I can run 8 jetty/tomcats and accomplish my goal but that is a
>>>> maintenance headache and is really a last resort. I really would just
>>>> like to be able to deploy this big machine with 'numShards=8'.
>>>>
>>>> Is that possible or do I really need to have 8 other nodes running?
>>>
>>>
>>> You don't want to run more than one container or Solr instance per
>>> machine. Things can get very confused, and it's too much overhead.
>>
>>>
>>>
>>> With existing collections, you can simply run the CoreAdmin CREATE
>>>
>>> action on the new node with more resources.
>>>
>>> So you'd do something like this, once for each of the 8 existing parts:
>>>
>>>
>>> http://newnode:port/solr/admin/cores?action=CREATE&name=collname_shard1_replica2&collection=collname&shard=shard1
>>>
>>> It will automatically replicate the shard from its current leader.
>>
>>
>> Fantastic! Clearly my understanding of "collection", vs "core" vs "shard"
>> was lacking but now I see the relationship better.
>>
>>
>>>
>>> One thing to be aware of: With 1.4TB of index data, it might be
>>> impossible to keep enough of the index in RAM for good performance,
>>> unless the machine has a terabyte or more of RAM.
>>
>>
>> Yes, I'm well aware of the performance implications, many of which are
>> mitigated by 2TB of SSD and 512GB RAM.
>>
>> Thanks for the nudge in the right direction. The first node/shard1 is
>> replicating right now.
>>
>> David
>>
>>
>>
Re: adding a node to SolrCloud
Posted by Greg Preston <gp...@marinsoftware.com>.
>Yes, I'm well aware of the performance implications, many of which are mitigated by 2TB of SSD and 512GB RAM
I've got a very similar setup in production. 2TB SSD, 256G RAM (128G
heaps), and 1 - 1.5 TB of index per node. We're in the process of
splitting that to multiple JVMs per host. GC pauses were causing ZK
timeouts (you should up that in solr.xml). And resync's after the
timeouts took long enough that a large tlog built up (we have near
continuous indexing), and we couldn't replay the tlog fast enough to
catch up to current.
If you're going to have a mostly static index, then it may be less of an issue.
-Greg
On Mon, Dec 23, 2013 at 2:31 AM, David Santamauro
<da...@gmail.com> wrote:
> On 12/22/2013 09:48 PM, Shawn Heisey wrote:
>>
>> On 12/22/2013 2:10 PM, David Santamauro wrote:
>>>
>>> My goal is to have a redundant copy of all 8 currently running, but
>>> non-redundant shards. This setup (8 nodes with no replicas) was a test
>>> and it has proven quite functional from a performance perspective.
>>> Loading, though, takes almost 3 weeks so I'm really not in a position to
>>> redesign the distribution, though I can add nodes.
>>>
>>> I have acquired another resource, a very large machine that I'd like to
>>> use to hold the replicas of the currently deployed 8-nodes.
>>>
>>> I realize I can run 8 jetty/tomcats and accomplish my goal but that is a
>>> maintenance headache and is really a last resort. I really would just
>>> like to be able to deploy this big machine with 'numShards=8'.
>>>
>>> Is that possible or do I really need to have 8 other nodes running?
>>
>>
>> You don't want to run more than one container or Solr instance per
>> machine. Things can get very confused, and it's too much overhead.
>
>>
>>
>> With existing collections, you can simply run the CoreAdmin CREATE
>>
>> action on the new node with more resources.
>>
>> So you'd do something like this, once for each of the 8 existing parts:
>>
>>
>> http://newnode:port/solr/admin/cores?action=CREATE&name=collname_shard1_replica2&collection=collname&shard=shard1
>>
>> It will automatically replicate the shard from its current leader.
>
>
> Fantastic! Clearly my understanding of "collection", vs "core" vs "shard"
> was lacking but now I see the relationship better.
>
>
>>
>> One thing to be aware of: With 1.4TB of index data, it might be
>> impossible to keep enough of the index in RAM for good performance,
>> unless the machine has a terabyte or more of RAM.
>
>
> Yes, I'm well aware of the performance implications, many of which are
> mitigated by 2TB of SSD and 512GB RAM.
>
> Thanks for the nudge in the right direction. The first node/shard1 is
> replicating right now.
>
> David
>
>
>
Re: adding a node to SolrCloud
Posted by David Santamauro <da...@gmail.com>.
On 12/22/2013 09:48 PM, Shawn Heisey wrote:
> On 12/22/2013 2:10 PM, David Santamauro wrote:
>> My goal is to have a redundant copy of all 8 currently running, but
>> non-redundant shards. This setup (8 nodes with no replicas) was a test
>> and it has proven quite functional from a performance perspective.
>> Loading, though, takes almost 3 weeks so I'm really not in a position to
>> redesign the distribution, though I can add nodes.
>>
>> I have acquired another resource, a very large machine that I'd like to
>> use to hold the replicas of the currently deployed 8-nodes.
>>
>> I realize I can run 8 jetty/tomcats and accomplish my goal but that is a
>> maintenance headache and is really a last resort. I really would just
>> like to be able to deploy this big machine with 'numShards=8'.
>>
>> Is that possible or do I really need to have 8 other nodes running?
>
> You don't want to run more than one container or Solr instance per
> machine. Things can get very confused, and it's too much overhead.
>
> With existing collections, you can simply run the CoreAdmin CREATE
> action on the new node with more resources.
>
> So you'd do something like this, once for each of the 8 existing parts:
>
> http://newnode:port/solr/admin/cores?action=CREATE&name=collname_shard1_replica2&collection=collname&shard=shard1
>
> It will automatically replicate the shard from its current leader.
Fantastic! Clearly my understanding of "collection", vs "core" vs
"shard" was lacking but now I see the relationship better.
>
> One thing to be aware of: With 1.4TB of index data, it might be
> impossible to keep enough of the index in RAM for good performance,
> unless the machine has a terabyte or more of RAM.
Yes, I'm well aware of the performance implications, many of which are
mitigated by 2TB of SSD and 512GB RAM.
Thanks for the nudge in the right direction. The first node/shard1 is
replicating right now.
David
Re: adding a node to SolrCloud
Posted by Shawn Heisey <so...@elyograg.org>.
On 12/22/2013 2:10 PM, David Santamauro wrote:
> My goal is to have a redundant copy of all 8 currently running, but
> non-redundant shards. This setup (8 nodes with no replicas) was a test
> and it has proven quite functional from a performance perspective.
> Loading, though, takes almost 3 weeks so I'm really not in a position to
> redesign the distribution, though I can add nodes.
>
> I have acquired another resource, a very large machine that I'd like to
> use to hold the replicas of the currently deployed 8-nodes.
>
> I realize I can run 8 jetty/tomcats and accomplish my goal but that is a
> maintenance headache and is really a last resort. I really would just
> like to be able to deploy this big machine with 'numShards=8'.
>
> Is that possible or do I really need to have 8 other nodes running?
You don't want to run more than one container or Solr instance per
machine. Things can get very confused, and it's too much overhead.
Also, you shouldn't start Solr with the numShards parameter on the
commandline. That should be given when you create each collection.
With existing collections, you can simply run the CoreAdmin CREATE
action on the new node with more resources.
http://wiki.apache.org/solr/SolrCloud#Creating_cores_via_CoreAdmin
So you'd do something like this, once for each of the 8 existing parts:
http://newnode:port/solr/admin/cores?action=CREATE&name=collname_shard1_replica2&collection=collname&shard=shard1
It will automatically replicate the shard from its current leader.
One thing to be aware of: With 1.4TB of index data, it might be
impossible to keep enough of the index in RAM for good performance,
unless the machine has a terabyte or more of RAM.
http://wiki.apache.org/solr/SolrPerformanceProblems#OS_Disk_Cache
Thanks,
Shawn
Re: adding a node to SolrCloud
Posted by David Santamauro <da...@gmail.com>.
Thanks for the reply.
My goal is to have a redundant copy of all 8 currently running, but
non-redundant shards. This setup (8 nodes with no replicas) was a test
and it has proven quite functional from a performance perspective.
Loading, though, takes almost 3 weeks so I'm really not in a position to
redesign the distribution, though I can add nodes.
I have acquired another resource, a very large machine that I'd like to
use to hold the replicas of the currently deployed 8-nodes.
I realize I can run 8 jetty/tomcats and accomplish my goal but that is a
maintenance headache and is really a last resort. I really would just
like to be able to deploy this big machine with 'numShards=8'.
Is that possible or do I really need to have 8 other nodes running?
David
On 12/22/2013 03:58 PM, Furkan KAMACI wrote:
> Hi David;
>
> When you start up 8 nodes within that machine they will be replicas of each
> shards and you will accomplish what you want. However if you can give more
> detail about your hardware infrastructure and needs I can offer you a
> design.
>
> Thanks;
> Furkan KAMACI
>
>
> 22 Aralık 2013 Pazar tarihinde David Santamauro <da...@gmail.com>
> adlı kullanıcı şöyle yazdı:
>>
>> any hint?
>>
>> On 12/22/2013 06:48 AM, David Santamauro wrote:
>>>
>>> Hi,
>>>
>>> I have an 8-node setup currently with 1 shard per node (no redundancy).
>>> These 8 nodes are smaller machines not capable of supporting the entire
>>> collection..
>>>
>>> I have another machine resource that can act as other node and this last
>>> node is capable of holding the entire collection. I'd like to make this
>>> large node hold a replica of all the other 8 nodes' shards for
>>> redundancy as well as performance.
>>>
>>> Is this possible? ... and what would be the configuration magic needed
>>> to accomplish this? This collection is very large (1.4TB) and as you can
>>> imagine, I don't want to fiddle around trying configuration alternatives
>>> assuming each try would lead to a replication of 1.4TB of data.
>>>
>>> Any help would be greatly appreciated.
>>>
>>> David
>>
>>
>
Re: adding a node to SolrCloud
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi David;
When you start up 8 nodes within that machine they will be replicas of each
shards and you will accomplish what you want. However if you can give more
detail about your hardware infrastructure and needs I can offer you a
design.
Thanks;
Furkan KAMACI
22 Aralık 2013 Pazar tarihinde David Santamauro <da...@gmail.com>
adlı kullanıcı şöyle yazdı:
>
> any hint?
>
> On 12/22/2013 06:48 AM, David Santamauro wrote:
>>
>> Hi,
>>
>> I have an 8-node setup currently with 1 shard per node (no redundancy).
>> These 8 nodes are smaller machines not capable of supporting the entire
>> collection..
>>
>> I have another machine resource that can act as other node and this last
>> node is capable of holding the entire collection. I'd like to make this
>> large node hold a replica of all the other 8 nodes' shards for
>> redundancy as well as performance.
>>
>> Is this possible? ... and what would be the configuration magic needed
>> to accomplish this? This collection is very large (1.4TB) and as you can
>> imagine, I don't want to fiddle around trying configuration alternatives
>> assuming each try would lead to a replication of 1.4TB of data.
>>
>> Any help would be greatly appreciated.
>>
>> David
>
>
Re: adding a node to SolrCloud
Posted by David Santamauro <da...@gmail.com>.
any hint?
On 12/22/2013 06:48 AM, David Santamauro wrote:
>
> Hi,
>
> I have an 8-node setup currently with 1 shard per node (no redundancy).
> These 8 nodes are smaller machines not capable of supporting the entire
> collection..
>
> I have another machine resource that can act as other node and this last
> node is capable of holding the entire collection. I'd like to make this
> large node hold a replica of all the other 8 nodes' shards for
> redundancy as well as performance.
>
> Is this possible? ... and what would be the configuration magic needed
> to accomplish this? This collection is very large (1.4TB) and as you can
> imagine, I don't want to fiddle around trying configuration alternatives
> assuming each try would lead to a replication of 1.4TB of data.
>
> Any help would be greatly appreciated.
>
> David