You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by adityab <ad...@yahoo.com> on 2012/11/01 21:46:39 UTC
Production Release process with Solr 3.5 implementation.
Hi,
My question is generally to understand how people are dealing with such
situation to minimize production downtime. Any suggestion/advice are
welcomed.
We have Solr 3.5 setup as master Slave. Our UI almost has 70% data fetched
from Solr Slaves where as other 30% are for authentication and other
validations where it needs to hit DB.
Index size is 20GB, # docs is 20M+
We have version upgrade like every 3 month coz of new features. If we make
any schema change for adding new fields or removing fields from Index or
changing Analyzers, we need to rebuild the index. We follow below steps.
1. Pull up Maintenance Pages
2. Upgrade DB
3. Upgrade Solr Schema (Master) Replication is disabled.
4. Start Index Rebuild. (if step 3)
5. Upgrade UI code
6. Index build complete ? Start Replication
7. Verify UI and Drop Maintenance Pages.
As # 4 takes couple of hours compared to all other steps which run within
few minutes, we need to have down time for the duration of that.
Our challenge is to reduce the downtime to least possible. Making UI
compatible with old version is not always possible.
Any suggestions/advice would be a great help. Let me know if more
information is required.
thanks
Aditya
--
View this message in context: http://lucene.472066.n3.nabble.com/Production-Release-process-with-Solr-3-5-implementation-tp4017679.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Production Release process with Solr 3.5 implementation.
Posted by adityab <ad...@yahoo.com>.
Thanks Paul,
In our case we need to upgrade DB with new version before re-building the
index as we pull data from DB.
Once DB is upgraded, UI needs to be upgraded too.
We do the same in case when there is no DB schema changes. We disable the
replication and build index on Master. Once index is ready we upgrade UI,
enable replication, verify Ui and drop maintenance page. This is like doing
the background indexing.
But when we have Schema update for both DB and Solr, this is where we run
into longer downtime.
--
View this message in context: http://lucene.472066.n3.nabble.com/Production-Release-process-with-Solr-3-5-implementation-tp4017679p4017693.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Production Release process with Solr 3.5 implementation.
Posted by Paul Libbrecht <pa...@hoplahup.net>.
Here is what we do for Curriki.org:
We run a background indexing by setting up another solr that performs all the indexing.
Then we can start the install process.
Then we can update the index with the things changed since the background indexing.
paul
Le 1 nov. 2012 à 21:46, adityab a écrit :
> Hi,
> My question is generally to understand how people are dealing with such
> situation to minimize production downtime. Any suggestion/advice are
> welcomed.
>
> We have Solr 3.5 setup as master Slave. Our UI almost has 70% data fetched
> from Solr Slaves where as other 30% are for authentication and other
> validations where it needs to hit DB.
> Index size is 20GB, # docs is 20M+
>
> We have version upgrade like every 3 month coz of new features. If we make
> any schema change for adding new fields or removing fields from Index or
> changing Analyzers, we need to rebuild the index. We follow below steps.
>
> 1. Pull up Maintenance Pages
> 2. Upgrade DB
> 3. Upgrade Solr Schema (Master) Replication is disabled.
> 4. Start Index Rebuild. (if step 3)
> 5. Upgrade UI code
> 6. Index build complete ? Start Replication
> 7. Verify UI and Drop Maintenance Pages.
>
> As # 4 takes couple of hours compared to all other steps which run within
> few minutes, we need to have down time for the duration of that.
>
> Our challenge is to reduce the downtime to least possible. Making UI
> compatible with old version is not always possible.
>
> Any suggestions/advice would be a great help. Let me know if more
> information is required.
>
> thanks
> Aditya
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Production-Release-process-with-Solr-3-5-implementation-tp4017679.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Production Release process with Solr 3.5 implementation.
Posted by adityab <ad...@yahoo.com>.
Our main issue is for new index we need to update DB first as data needs to
be pulled from DB as per new schema and publish to Solr.
So DB update has to be first step.
--
View this message in context: http://lucene.472066.n3.nabble.com/Production-Release-process-with-Solr-3-5-implementation-tp4017679p4017740.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Production Release process with Solr 3.5 implementation.
Posted by Alexander Kanarsky <ka...@gmail.com>.
Why not to change the order to this:
3. Upgrade Solr Schema (Master) Replication is disabled.
4. Start Index Rebuild. (if step 3)
1. Pull up Maintenance Pages
2. Upgrade DB
5. Upgrade UI code
6. Index build complete ? Start Replication
7. Verify UI and Drop Maintenance Pages.
So your slaves will continue to serve traffic until you're done with the
master index. Or the master index also imports from the same database?
On Thu, Nov 1, 2012 at 4:08 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 11/1/2012 2:46 PM, adityab wrote:
>
>> 1. Pull up Maintenance Pages
>> 2. Upgrade DB
>> 3. Upgrade Solr Schema (Master) Replication is disabled.
>> 4. Start Index Rebuild. (if step 3)
>> 5. Upgrade UI code
>> 6. Index build complete ? Start Replication
>> 7. Verify UI and Drop Maintenance Pages.
>>
>> As # 4 takes couple of hours compared to all other steps which run within
>> few minutes, we need to have down time for the duration of that.
>>
>
> What I do is a little bit different. I have two completely independent
> copies of my index, no replication. The build system maintains each copy
> simultaneously, including managing independent rebuilds. I used to run two
> copies of my build system, but I recently made it so that one copy manages
> multiple indexes.
>
> If I need to do an upgrade, I will first test everything out as much as
> possible on my test environment. Then I will take one copy of my index
> offline, perform the required changes, and reindex. The UI continues to
> send queries to the online index that hasn't been changed. At that point,
> we initiate the upgrade sequence you've described, except that instead of
> step 4 taking a few hours, we just have to redirect traffic to the brand
> new index copy. If everything works out, we then repeat with the other
> index copy. If it doesn't work out, we revert everything and go back to
> the original index.
>
> Also, every index has a build core and a live core. I currently maintain
> the same config in both cores, but it would be possible to change the
> config in the build core, reload or restart Solr, do your reindex, and
> simply do a core swap, which is almost instantaneous. If you are doing
> replication, swapping cores on the master initiates full replication to the
> slave. Excerpt from my solr.xml:
>
> <core instanceDir="cores/s0_1/" name="s0live"
> dataDir="../../data/s0_1"/>
> <core instanceDir="cores/s0_0/" name="s0build"
> dataDir="../../data/s0_0"/>
>
> Thanks,
> Shawn
>
> P.S. Actually, I have three full copies of my index now -- I recently
> upgraded my test server so it has enough disk capacity to hold my entire
> index. The test server runs a local copy of the build system which keeps
> it up to date with the two production copies.
>
>
Re: Production Release process with Solr 3.5 implementation.
Posted by Shawn Heisey <so...@elyograg.org>.
On 11/1/2012 2:46 PM, adityab wrote:
> 1. Pull up Maintenance Pages
> 2. Upgrade DB
> 3. Upgrade Solr Schema (Master) Replication is disabled.
> 4. Start Index Rebuild. (if step 3)
> 5. Upgrade UI code
> 6. Index build complete ? Start Replication
> 7. Verify UI and Drop Maintenance Pages.
>
> As # 4 takes couple of hours compared to all other steps which run within
> few minutes, we need to have down time for the duration of that.
What I do is a little bit different. I have two completely independent
copies of my index, no replication. The build system maintains each
copy simultaneously, including managing independent rebuilds. I used to
run two copies of my build system, but I recently made it so that one
copy manages multiple indexes.
If I need to do an upgrade, I will first test everything out as much as
possible on my test environment. Then I will take one copy of my index
offline, perform the required changes, and reindex. The UI continues to
send queries to the online index that hasn't been changed. At that
point, we initiate the upgrade sequence you've described, except that
instead of step 4 taking a few hours, we just have to redirect traffic
to the brand new index copy. If everything works out, we then repeat
with the other index copy. If it doesn't work out, we revert everything
and go back to the original index.
Also, every index has a build core and a live core. I currently
maintain the same config in both cores, but it would be possible to
change the config in the build core, reload or restart Solr, do your
reindex, and simply do a core swap, which is almost instantaneous. If
you are doing replication, swapping cores on the master initiates full
replication to the slave. Excerpt from my solr.xml:
<core instanceDir="cores/s0_1/" name="s0live"
dataDir="../../data/s0_1"/>
<core instanceDir="cores/s0_0/" name="s0build"
dataDir="../../data/s0_0"/>
Thanks,
Shawn
P.S. Actually, I have three full copies of my index now -- I recently
upgraded my test server so it has enough disk capacity to hold my entire
index. The test server runs a local copy of the build system which
keeps it up to date with the two production copies.