You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by KeesSchepers <ke...@keesschepers.nl> on 2012/03/14 19:58:30 UTC
Solr core swap after rebuild in HA-setup / High-traffic
Hello everybody,
I am designing a new Solr architecture for one of my clients. This sorl
architecture is for a high-traffic website with million of visitors but I am
facing some design problems were I hope you guys could help me out.
In my situation there are 4 Solr servers running, 1 server is master and 3
are slave. They are running Solr version 1.4.
I use two cores 'live' and 'rebuild' and I use Solr DIH to rebuild a core
which goes like this:
1. I wipe the reindex core
2. I run the DIH to the complete dataset (4 million documents) in peices of
20.000 records (to prevent very long mysql locks)
3. After the DIH is finished (2 hours) we have to also have to update the
rebuild core with changes from the last two hours, this is a problem
4. After updating is done and the core is not more then some seconds behind
we want to SWAP the cores.
Everything goes well except for step 3. The rebuild and the core swap is all
okay.
Because the website is undergoing changes every minute we cannot pauze the
delta-import on the live and walk behind for 2 hours. The problem is that I
can't figure out a closing system with not delaying the live core to long
and use the DIH instead of writing a lot of code.
Did anyone face this problem before or could give me some tips?
Thanks!
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-core-swap-after-rebuild-in-HA-setup-High-traffic-tp3826461p3826461.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr core swap after rebuild in HA-setup / High-traffic
Posted by KeesSchepers <ke...@keesschepers.nl>.
Well, the point is as follows.
We have a mysql table where all the changes are tracked something very
simular to your situation. The first problem is that, the delta-import on
the live core needs to update this table to notify a record is done. I do
this very awfull now within a script transformer, offcourse DIH isn't
designed for this.
The second thing is, that if the rebuild is running on the rebuild core, we
want to do a delta-import on this new core to make it less behind from the
live core but also while the rebuilding process is ongoing also the
delta-import on the live core runs every minute.
The second problem is that the delta-import on the live core already set's
these rows to status 'processed' and the delta-update after the rebuild
wouldn't pick up these updates anymore.
There are some solutions but I can't figure out a clean way to solve this
architecture problem. Maybe there isn't a clean solution..
I am curious how other developers experiencing this thing..
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-core-swap-after-rebuild-in-HA-setup-High-traffic-tp3826461p3826835.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr core swap after rebuild in HA-setup / High-traffic
Posted by Bill Bell <bi...@gmail.com>.
DIH sets the time of update to the start time not the end time,
So when the index is rebuilt, if you run an delta and use the update time you should be okay. We normally go back a few minutes to make sure we have all s a fail safe as well.
Sent from my Mobile device
720-256-8076
On Mar 14, 2012, at 12:58 PM, KeesSchepers <ke...@keesschepers.nl> wrote:
> Hello everybody,
>
> I am designing a new Solr architecture for one of my clients. This sorl
> architecture is for a high-traffic website with million of visitors but I am
> facing some design problems were I hope you guys could help me out.
>
> In my situation there are 4 Solr servers running, 1 server is master and 3
> are slave. They are running Solr version 1.4.
>
> I use two cores 'live' and 'rebuild' and I use Solr DIH to rebuild a core
> which goes like this:
>
> 1. I wipe the reindex core
> 2. I run the DIH to the complete dataset (4 million documents) in peices of
> 20.000 records (to prevent very long mysql locks)
> 3. After the DIH is finished (2 hours) we have to also have to update the
> rebuild core with changes from the last two hours, this is a problem
> 4. After updating is done and the core is not more then some seconds behind
> we want to SWAP the cores.
>
> Everything goes well except for step 3. The rebuild and the core swap is all
> okay.
>
> Because the website is undergoing changes every minute we cannot pauze the
> delta-import on the live and walk behind for 2 hours. The problem is that I
> can't figure out a closing system with not delaying the live core to long
> and use the DIH instead of writing a lot of code.
>
> Did anyone face this problem before or could give me some tips?
>
> Thanks!
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-core-swap-after-rebuild-in-HA-setup-High-traffic-tp3826461p3826461.html
> Sent from the Solr - User mailing list archive at Nabble.com.
RE: Solr core swap after rebuild in HA-setup / High-traffic
Posted by "Young, Cody" <Co...@move.com>.
We have a very similar system. In our case we have a row version field in our sql database. When we run the full import we keep track of the latest row version at the time that the full import started. Once the full import is done we run an optimize and then run a delta import (actually this is a full import in a different DIH file, same mappings just with some different parameters to the stored procedure that we call). This catches the offline core up to live and then we swap.
Why can't you run the delta import on the rebuild core? Is there some specific issue that is preventing you?
Cody
-----Original Message-----
From: KeesSchepers [mailto:kees@keesschepers.nl]
Sent: Wednesday, March 14, 2012 11:59 AM
To: solr-user@lucene.apache.org
Subject: Solr core swap after rebuild in HA-setup / High-traffic
Hello everybody,
I am designing a new Solr architecture for one of my clients. This sorl architecture is for a high-traffic website with million of visitors but I am facing some design problems were I hope you guys could help me out.
In my situation there are 4 Solr servers running, 1 server is master and 3 are slave. They are running Solr version 1.4.
I use two cores 'live' and 'rebuild' and I use Solr DIH to rebuild a core which goes like this:
1. I wipe the reindex core
2. I run the DIH to the complete dataset (4 million documents) in peices of
20.000 records (to prevent very long mysql locks) 3. After the DIH is finished (2 hours) we have to also have to update the rebuild core with changes from the last two hours, this is a problem 4. After updating is done and the core is not more then some seconds behind we want to SWAP the cores.
Everything goes well except for step 3. The rebuild and the core swap is all okay.
Because the website is undergoing changes every minute we cannot pauze the delta-import on the live and walk behind for 2 hours. The problem is that I can't figure out a closing system with not delaying the live core to long and use the DIH instead of writing a lot of code.
Did anyone face this problem before or could give me some tips?
Thanks!
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-core-swap-after-rebuild-in-HA-setup-High-traffic-tp3826461p3826461.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr core swap after rebuild in HA-setup / High-traffic
Posted by Shawn Heisey <so...@elyograg.org>.
On 3/14/2012 12:58 PM, KeesSchepers wrote:
> 1. I wipe the reindex core
> 2. I run the DIH to the complete dataset (4 million documents) in peices of
> 20.000 records (to prevent very long mysql locks)
> 3. After the DIH is finished (2 hours) we have to also have to update the
> rebuild core with changes from the last two hours, this is a problem
> 4. After updating is done and the core is not more then some seconds behind
> we want to SWAP the cores.
>
> Everything goes well except for step 3. The rebuild and the core swap is all
> okay.
>
> Because the website is undergoing changes every minute we cannot pauze the
> delta-import on the live and walk behind for 2 hours. The problem is that I
> can't figure out a closing system with not delaying the live core to long
> and use the DIH instead of writing a lot of code.
I solve this problem by tracking the current position outside of the
database and Solr, in my build system. The primary key on the mysql
table is an autoincrement BIGINT field - I just keep track of the last
value that was added. When a rebuild happens, I continue to track it
for the live cores, then when it comes time to swap, I restore the
tracked value to what it was when the rebuild started. You can put
arbitrary parameters on your DIH url and have DIH construct your query
with them.
I used to have a build system written in Perl that did *all* index
activity with DIH. Now I have a Java build system that only uses DIH
for full index rebuilds. It uses SolrJ for everything else.
Thanks,
Shawn