You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by rabahb <ra...@gmail.com> on 2010/05/18 10:22:22 UTC

Solr Architecture discussion

Hi,

I'd like to get some architectural advices concerning the setup of a solr
(v1.4) platform in a production environment. 
I'll first describe my targeted architecture and then ask the questions
related to that environment.


Here's briefly what I achieved so far:

I've already setup an environment which serves as a proof of concept. This
environment is composed of a master instance on one host, 
and a slave instance on a second host. The slave handles 2 solr cores. 
In the final version of the architecture I would add up one ore more SLAVE
nodes depending on the request load.  

                                                  request   
                                                       |
                                                       V  
[  MASTER [core]  ] ------- [SLAVE [core1] <--swap-->[core2]  ]
                   |
                   v
           [index backup]

The goal of this architecture is:
* Isolate indexing from requesting
* Enable index replication from master to slave
* Control the swap between newly replicated index  (use of dual core per
Slave ) 

Here's how the whole platform works when we need to renew the index (on the
slaves)
1- backup index files on master using solr backup capability (a backup is
always welcome)
2- launch index creation (I'm using the delta indexing capabilities in order
to limit the index generation time)
3- trigger replication from master core to slave core2 based on solr
capabilities too
4- trigger swap between core 1 and core2
5- At this point Slave index has been renewed ... we can revert back to the
previous index if there was any issues with the new one.

As this is aimed to be a production environment, redondancy is one of the
key elements, meaning that will double (or more) the front solr 
instances. If slave instances are not in the same network as the Master
instance, our strategy will probably be to set up one of the slaves 
as a relay.

That said, here are my questions:

1 /  I'd like to have insight about issues that may happen with that kind of
architecture?

2 / My first concern is about the size of the index that would need to be
replicated. We need to perform indexing all day long (every 5min) and
replicate as soon as the index is built.
As far as I know, replication copies over all the index files. I think that
there can not be delta replication (only replicating what changed). That's
my assumption. 
But, is there any way to make a delta replication if that make any sense?

3 / How can I improve this architecture based on your own experience?
Ex: Shall I use different network interface for solr commands and requests?

Thank you for sharing.
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Architecture-discussion-tp825708p825708.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Architecture discussion

Posted by rabahb <ra...@gmail.com>.
Do you have any insights that could help me and other people that might be
interested in that discussion?
Thanks.

-- 
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Architecture-discussion-tp825708p828658.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Architecture discussion

Posted by Chris Hostetter <ho...@fucit.org>.
: B- A backup of the current index would be created
: C- Re-Indexing will happen on Master-core2 
: D- When Indexing is done, we'll trigger a swap between Master-core1 and
: core2
	...
: But how can B,C, and D. I'll do it manually. Wait! I'm not sure my boss will
: pay for that.

: 1/Can I leverage on some solr mechanisms (that is, by configuration only) in
: order to reach that goal?
: I haven't found how to do it!

your best bet is some external scheduler -- depending on how your build 
process works, you can fairly easily integrate it into external 
publishing tools.

: 2/ Is there any issue while replicating master "swapped" index files? I've
: seen in the literature that there might be some issues.

As long as the "new" version of the index is treuly "newer" then the old 
version, there shouldn't be any problem.

Frankly though: i'm not sure you need core swapping on the master either 
-- it depends largely on how much "churn" will happen each time you do one 
of these full rebuilds.  you could just as easily do incremental 
reindexing on your master, with occasional commits (or even autocommits) 
nad your slaves picking up those new segments -- either gradually, or all 
at once when you do a monolithic commit. 

if you're ok with the slaves pulling over the *entire* index after you do 
the core swap, then you should be fine with the slaves pulling over the 
*entire* index (or maybe just most of it) after a rebuild directly to the 
existing core.

all you really need to do explicitly on the master is trigger a backup 
just before you rebuild the world, and if (and only if) something goes 
terribly wrong, then restore from your backup.




-Hoss


Re: Solr Architecture discussion

Posted by rabahb <ra...@gmail.com>.
Thinking twice about this architecture ...

I'm concerned about the way I'm going to automate the following steps:

A- The slaves would regularly poll Master-core1 for changes
B- A backup of the current index would be created
C- Re-Indexing will happen on Master-core2 
D- When Indexing is done, we'll trigger a swap between Master-core1 and
core2
E- Slaves will then poll and pickup the freshly updated index segments
F- and so on!

This seems to be simple when it's done manually. But I can not just sit
there and trigger a button to send the events. To reach that goal, I
realized that on solution would be to have 2 cores on the master side, while
the slaves would only have one core (as previously discussed). We'll just
need to configure the slave polling period (A,E), and send the right http
request (B,C,D). 

Well ok, step A is automated "natively". Easy enough, using the internal
solr capabilities.
But how can B,C, and D. I'll do it manually. Wait! I'm not sure my boss will
pay for that.

All right so I imagine that I should implement a process that will automate
the phases that I would otherwise do manually. This would be an external
process not based on solr mechanism.

My questions are:

1/Can I leverage on some solr mechanisms (that is, by configuration only) in
order to reach that goal?
I haven't found how to do it!

2/ Is there any issue while replicating master "swapped" index files? I've
seen in the literature that there might be some issues.

3/If a solr configuration based solution does not exist, my first attempt
would be to write a shell based process that will regularly trigger the
events, wait for the end of each phase by polling the current phase status,
in order to trigger the next one. Does that sound good to you? Or is there a
better and more elegant way to do the trick when indexing and replication
should be beating at a high pace? 

Thank you.
 

-- 
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Architecture-discussion-tp825708p860942.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Architecture discussion

Posted by rabahb <ra...@gmail.com>.
Hi Chris,

Thanks for your insights. I totally understand your point about steps 4 and
5. I wanted to control the moment when the swap would happen on the slave
side but as you say there is no use for that. It only adds up complexity
that internal solr mechanisms are already providing.  

For the replication aspect, I re-read the whole documentation and with the
light you shed on that topic, I realize that the only problem here is the
huge amount of data that can be passed over the wire depending on the
segments that the indexing will update. As you say, optimizing can have a
devastating effect on the replication phase as, if I have a good
understanding of what you said, this could potentially update all the index
segments. 

OK! so if I rephrase it, the best strategy in my case is to limit the
optimization phases in order to prioritize the replication performance, and
make the optimization only when the replication activity is not so crucial
in order to avoid degrading the search performances. 

Thank you very much. That helps a lot.






-- 
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Architecture-discussion-tp825708p860767.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Architecture discussion

Posted by Chris Hostetter <ho...@fucit.org>.
: 4- trigger swap between core 1 and core2
: 5- At this point Slave index has been renewed ... we can revert back to the
: previous index if there was any issues with the new one.

these steps are largely unneccessary -- within a single SolrCore Solr 
already keeps track of the "current" searcher (which is serving requests) 
and one or more "onDeck" searchers which know about "newer" versions of 
the index -- it can manage warming up caches for hte "onDeck" seracher for 
you (and can do it more effectively then you can becuase it can do so 
based on the contents of the "old" caches from the "current" searcher).

While there may be some value in being able to "revert" on the slaves, 
this is typically just as easy by configuring it to keep snapshots arround 
for a while and if you have a problem manually restore from one of hte 
older snapshots -- this will tide you over until you fix whatever the 
problem is on the master, possibly by restoring from a backup, and then 
start replication again.

: 2 / My first concern is about the size of the index that would need to be
: replicated. We need to perform indexing all day long (every 5min) and
: replicate as soon as the index is built.
: As far as I know, replication copies over all the index files. I think that
: there can not be delta replication (only replicating what changed). That's
: my assumption. 
: But, is there any way to make a delta replication if that make any sense?

replication does *not* copy over all of hte index files.  replication 
"syncs" all of hte index files, but only data is sent over the wire -- in 
cases of segment merges (or full optimizations) this can result in 
fluctuations and "higher then typical" amounts of data getting sent over 
the wire even when the only a few docs have been added/deleted, but this 
is easy to manage.  (don't optimize during times when replication speed is 
critical, and use conservative merge settings)



-Hoss