You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mike Lissner <ml...@michaeljaylissner.com> on 2017/06/14 18:26:12 UTC
Swapping indexes on disk
We are replacing a drive mounted at /old with one mounted at /new. Our
index currently lives on /old, and our plan was to:
1. Create a new index on /new
2. Reindex from our database so that the new index on /new is properly
populated.
3. Stop solr.
4. Symlink /old to /new (Solr now looks for the index at /old/solr, which
redirects to /new/solr)
5. Start solr
6. (Later) Stop solr, swap the drives (old for new), and start solr. (Solr
now looks for the index at /old/solr again, and finds it there.)
7. Delete the index pointing to /new created in step 1.
The idea was that this would create a new index for solr, would populate it
with the right content, and would avoid having to touch our existing solr
configurations aside from creating one new index, which we could soon
delete.
I just did steps 1-5, but I got null pointer exceptions when starting solr,
and it appears that the index on /new has been almost completely deleted by
Solr (this is a bummer, since it takes days to populate).
Is this expected? Am I terribly crazy to try to swap indexes on disk? As
far as I know, the only difference between the indexes is their name.
We're using Solr version 4.10.4.
Thank you,
Mike
Re: Swapping indexes on disk
Posted by Mike Lissner <ml...@michaeljaylissner.com>.
Weirdly, this happened again today, deleting a brand new 300GB index that
we created after last time, and which had been working for several days.
This time, the index was deleted by our log rotation script, which
restarted solr, so it's very easy to see what happened before and after the
problem. Before the restart, queries were working just fine, and then
during shutdown, the only problematic line is the last, which shows:
590125499 [Thread-0] WARN org.eclipse.jetty.util.thread.QueuedThreadPool
– 1 threads could not be stopped
590125500 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– Couldn't stop Thread[qtp597653135-10081,5,main]
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.lucene.search.FieldValueHitQueue$OneComparatorFieldValueHitQueue.lessThan(FieldValueHitQueue.java:84)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.lucene.search.FieldValueHitQueue$OneComparatorFieldValueHitQueue.lessThan(FieldValueHitQueue.java:58)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:246)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at org.apache.lucene.util.PriorityQueue.pop(PriorityQueue.java:177)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.lucene.search.TopFieldCollector.populateResults(TopFieldCollector.java:1207)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.java:156)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1622)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1433)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:514)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:484)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
590125501 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at org.eclipse.jetty.server.Server.handle(Server.java:368)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
590125502 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
590125503 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
590125503 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
590125503 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
590125503 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
590125503 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
590125503 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
590125503 [Thread-0] INFO org.eclipse.jetty.util.thread.QueuedThreadPool
– at java.lang.Thread.run(Thread.java:745)
Those are the last lines in the log, after all of the other indexes shut
down properly.
After that, a new log file is started, and it cannot start the index,
complaining about missing files. So at that point, the index is gone.
I'd love to prevent this from happening a third time. It's super baffling.
Any ideas?
Mike
On Tue, Jun 20, 2017 at 12:38 PM Mike Lissner <
mlissner@michaeljaylissner.com> wrote:
> Thanks for the suggestions everybody.
>
> Some responses to Shawn's questions:
>
> > Does your solr.xml file contain core definitions, or is that information
> in a core.properties file in each instanceDir?
>
> Were using core.properties files.
>
> > How did you install Solr
>
> Solr is installed just by downloading and unzipping. From there, we use
> the example directories as a starting point.
>
>
> > and how are you starting it?
>
> Using a pretty simple init script. Nothing too exotic here.
>
> > Do you have the full error and stacktrace from those null pointer
> exceptions?
>
> I put a log of the startup here:
> https://www.courtlistener.com/tools/sample-data/misc/null_logs.txt
>
> I created this by doing `grep -C 1000 -i nullpointer`, then cleaning out
> any private queries. I looked through it a bit. It looks like the index was
> missing a file, and was therefore unable to start up. I won't say it's
> impossible that the index was deleted before I started Solr, but it seemed
> to be operating fine using the other name prior to stopping solr and
> putting in a symlink. In the real-world logs, our disks are named /sata and
> /sata8 instead of /old and /new.
>
>
> > In the context of that information, what exactly did you do at each step
> of your process?
>
> The process above was pretty boring really.
>
> 1. Create new index and populate it:
>
> - copied an existing index configuration into a new directory
> - tweaked the datadir parameter in core.properties
> - restarted solr
> - re-indexed the database using usual HTTP API to populate the new index
>
> 2. stop solr: sudo service solr stop
>
> 3. make symlink:
>
> - mv'ed the old index out of the way
> - ln -s old new (or vice versa, I never remember which way ln goes)
>
> 4. start solr: sudo service solr start
>
> FWIW, I've got it working now using the SWAP index functionality, so the
> above is just in case somebody wants to try to track this down. I'll
> probably take those logs offline after a week or two.
>
> Mike
>
>
> On Tue, Jun 20, 2017 at 7:20 AM Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 6/14/2017 12:26 PM, Mike Lissner wrote:
>> > We are replacing a drive mounted at /old with one mounted at /new. Our
>> > index currently lives on /old, and our plan was to:
>> >
>> > 1. Create a new index on /new
>> > 2. Reindex from our database so that the new index on /new is properly
>> > populated.
>> > 3. Stop solr.
>> > 4. Symlink /old to /new (Solr now looks for the index at /old/solr,
>> which
>> > redirects to /new/solr)
>> > 5. Start solr
>> > 6. (Later) Stop solr, swap the drives (old for new), and start solr.
>> (Solr
>> > now looks for the index at /old/solr again, and finds it there.)
>> > 7. Delete the index pointing to /new created in step 1.
>> >
>> > The idea was that this would create a new index for solr, would
>> populate it
>> > with the right content, and would avoid having to touch our existing
>> solr
>> > configurations aside from creating one new index, which we could soon
>> > delete.
>> >
>> > I just did steps 1-5, but I got null pointer exceptions when starting
>> solr,
>> > and it appears that the index on /new has been almost completely
>> deleted by
>> > Solr (this is a bummer, since it takes days to populate).
>> >
>> > Is this expected? Am I terribly crazy to try to swap indexes on disk? As
>> > far as I know, the only difference between the indexes is their name.
>> >
>> > We're using Solr version 4.10.4.
>>
>> Solr should not delete indexes on startup. The only time it should do
>> that is when you explicitly request deletion. Do you have the full
>> error and stacktrace from those null pointer exceptions? Something
>> would have to be very wrong for it to behave like you describe.
>>
>> Does your solr.xml file contain core definitions, or is that information
>> in a core.properties file in each instanceDir? The latter is the only
>> option supported in 5.0 and later, but the 4.10 version still supports
>> both.
>>
>> How is Solr itself and the data directories laid out? How did you
>> install Solr, and how are you starting it? In the context of that
>> information, what exactly did you do at each step of your process?
>>
>> Thanks,
>> Shawn
>>
>>
Re: Swapping indexes on disk
Posted by Mike Lissner <ml...@michaeljaylissner.com>.
Thanks for the suggestions everybody.
Some responses to Shawn's questions:
> Does your solr.xml file contain core definitions, or is that information
in a core.properties file in each instanceDir?
Were using core.properties files.
> How did you install Solr
Solr is installed just by downloading and unzipping. From there, we use the
example directories as a starting point.
> and how are you starting it?
Using a pretty simple init script. Nothing too exotic here.
> Do you have the full error and stacktrace from those null pointer
exceptions?
I put a log of the startup here:
https://www.courtlistener.com/tools/sample-data/misc/null_logs.txt
I created this by doing `grep -C 1000 -i nullpointer`, then cleaning out
any private queries. I looked through it a bit. It looks like the index was
missing a file, and was therefore unable to start up. I won't say it's
impossible that the index was deleted before I started Solr, but it seemed
to be operating fine using the other name prior to stopping solr and
putting in a symlink. In the real-world logs, our disks are named /sata and
/sata8 instead of /old and /new.
> In the context of that information, what exactly did you do at each step
of your process?
The process above was pretty boring really.
1. Create new index and populate it:
- copied an existing index configuration into a new directory
- tweaked the datadir parameter in core.properties
- restarted solr
- re-indexed the database using usual HTTP API to populate the new index
2. stop solr: sudo service solr stop
3. make symlink:
- mv'ed the old index out of the way
- ln -s old new (or vice versa, I never remember which way ln goes)
4. start solr: sudo service solr start
FWIW, I've got it working now using the SWAP index functionality, so the
above is just in case somebody wants to try to track this down. I'll
probably take those logs offline after a week or two.
Mike
On Tue, Jun 20, 2017 at 7:20 AM Shawn Heisey <ap...@elyograg.org> wrote:
> On 6/14/2017 12:26 PM, Mike Lissner wrote:
> > We are replacing a drive mounted at /old with one mounted at /new. Our
> > index currently lives on /old, and our plan was to:
> >
> > 1. Create a new index on /new
> > 2. Reindex from our database so that the new index on /new is properly
> > populated.
> > 3. Stop solr.
> > 4. Symlink /old to /new (Solr now looks for the index at /old/solr, which
> > redirects to /new/solr)
> > 5. Start solr
> > 6. (Later) Stop solr, swap the drives (old for new), and start solr.
> (Solr
> > now looks for the index at /old/solr again, and finds it there.)
> > 7. Delete the index pointing to /new created in step 1.
> >
> > The idea was that this would create a new index for solr, would populate
> it
> > with the right content, and would avoid having to touch our existing solr
> > configurations aside from creating one new index, which we could soon
> > delete.
> >
> > I just did steps 1-5, but I got null pointer exceptions when starting
> solr,
> > and it appears that the index on /new has been almost completely deleted
> by
> > Solr (this is a bummer, since it takes days to populate).
> >
> > Is this expected? Am I terribly crazy to try to swap indexes on disk? As
> > far as I know, the only difference between the indexes is their name.
> >
> > We're using Solr version 4.10.4.
>
> Solr should not delete indexes on startup. The only time it should do
> that is when you explicitly request deletion. Do you have the full
> error and stacktrace from those null pointer exceptions? Something
> would have to be very wrong for it to behave like you describe.
>
> Does your solr.xml file contain core definitions, or is that information
> in a core.properties file in each instanceDir? The latter is the only
> option supported in 5.0 and later, but the 4.10 version still supports
> both.
>
> How is Solr itself and the data directories laid out? How did you
> install Solr, and how are you starting it? In the context of that
> information, what exactly did you do at each step of your process?
>
> Thanks,
> Shawn
>
>
Re: Swapping indexes on disk
Posted by Shawn Heisey <ap...@elyograg.org>.
On 6/14/2017 12:26 PM, Mike Lissner wrote:
> We are replacing a drive mounted at /old with one mounted at /new. Our
> index currently lives on /old, and our plan was to:
>
> 1. Create a new index on /new
> 2. Reindex from our database so that the new index on /new is properly
> populated.
> 3. Stop solr.
> 4. Symlink /old to /new (Solr now looks for the index at /old/solr, which
> redirects to /new/solr)
> 5. Start solr
> 6. (Later) Stop solr, swap the drives (old for new), and start solr. (Solr
> now looks for the index at /old/solr again, and finds it there.)
> 7. Delete the index pointing to /new created in step 1.
>
> The idea was that this would create a new index for solr, would populate it
> with the right content, and would avoid having to touch our existing solr
> configurations aside from creating one new index, which we could soon
> delete.
>
> I just did steps 1-5, but I got null pointer exceptions when starting solr,
> and it appears that the index on /new has been almost completely deleted by
> Solr (this is a bummer, since it takes days to populate).
>
> Is this expected? Am I terribly crazy to try to swap indexes on disk? As
> far as I know, the only difference between the indexes is their name.
>
> We're using Solr version 4.10.4.
Solr should not delete indexes on startup. The only time it should do
that is when you explicitly request deletion. Do you have the full
error and stacktrace from those null pointer exceptions? Something
would have to be very wrong for it to behave like you describe.
Does your solr.xml file contain core definitions, or is that information
in a core.properties file in each instanceDir? The latter is the only
option supported in 5.0 and later, but the 4.10 version still supports both.
How is Solr itself and the data directories laid out? How did you
install Solr, and how are you starting it? In the context of that
information, what exactly did you do at each step of your process?
Thanks,
Shawn
Re: Swapping indexes on disk
Posted by Erick Erickson <er...@gmail.com>.
Why not just use the replication API and fetchindex? See:
https://cwiki.apache.org/confluence/display/solr/Index+Replication#IndexReplication-HTTPAPICommandsfortheReplicationHandler.
It's not entirely obvious from the writeup, but you can specify
masterUrl as part of the command &masterUrl=some_other_solr_core.
So you have your "live" core, and your just-indexed core, call it "new".
You issue the fetchindex command to core on live and masterUrl points
to "new". It'll look something like
http://liveserver:8983/solr/live_core/replication?command=fetchindex&masterUrl=http://newserver:8983/solr/new_core
Solr will
1> copy the index from new to live
2> once that's done, open a new searcher and you're searching live
documents (after any autowarming you've configured).
3> delete the old index
Note that until <2>, incoming searches are served by the old index. So
the user sees no service interruptions at all.
If for any reason the fetch fails, the old index is left intact.
This will require that you have enough disk space to temporarily have
both the old and new index on the live server and there may be some
extra memory consumed while the old searcher is still open and the new
one is autowarming.
Best,
Erick
On Wed, Jun 14, 2017 at 4:10 PM, Mike Lissner
<ml...@michaeljaylissner.com> wrote:
> I figured Solr would have a native system built in, but since we don't use
> it already, I didn't want to learn all of its ins and outs just for this
> disk situation.
>
> Ditto, essentially, applies for the swapping strategy. We don't have a Solr
> expert, just me, a generalist, and sorting out these kinds of things can
> take a while. The hope was to avoid that kind of complication with some
> clever use of symlinks and minor downtime. Our front end has a retry
> mechanism, so if solr is down for less than a minute, users will just have
> delayed responses, which is fine.
>
> The new strategy is to rsync the files while solr is live, stop solr, do a
> rsync diff, then start solr again. That'll give a bit for bit copy with
> very little downtime — it's the strategy postgres recommends for disk-based
> backups, so it seems like a safer bet. We needed a re-index anyway due to
> schema changes, which my first attempt included, but I guess that'll have
> to wait.
>
> Thanks for the replies. If anybody can explain why the first strategy
> failed, I'd still be interested in learning.
>
> Mike
>
> On Wed, Jun 14, 2017 at 12:09 PM Chris Ulicny <cu...@iq.media> wrote:
>
>> Are you physically swapping the disks to introduce the new index? Or having
>> both disks mounted at the same time?
>>
>> If the disks are simultaneously available, can you just swap the cores and
>> then delete the core on the old disk?
>>
>> https://cwiki.apache.org/confluence/display/solr/CoreAdmin+API#CoreAdminAPI-SWAP
>>
>> We periodically move cores to different drives using solr's replication
>> functionality and core swapping (after stopping replication). However, I've
>> never encountered solr deleting an index like that.
>>
>>
>>
>> On Wed, Jun 14, 2017 at 2:48 PM David Hastings <
>> hastings.recursive@gmail.com>
>> wrote:
>>
>> > I dont have an answer to why the folder got cleared, however i am
>> wondering
>> > why you arent using basic replication to do this exact same thing, since
>> > solr will natively take care of all this for you with no interruption to
>> > the user and no stop/start routines etc.
>> >
>> > On Wed, Jun 14, 2017 at 2:26 PM, Mike Lissner <
>> > mlissner@michaeljaylissner.com> wrote:
>> >
>> > > We are replacing a drive mounted at /old with one mounted at /new. Our
>> > > index currently lives on /old, and our plan was to:
>> > >
>> > > 1. Create a new index on /new
>> > > 2. Reindex from our database so that the new index on /new is properly
>> > > populated.
>> > > 3. Stop solr.
>> > > 4. Symlink /old to /new (Solr now looks for the index at /old/solr,
>> which
>> > > redirects to /new/solr)
>> > > 5. Start solr
>> > > 6. (Later) Stop solr, swap the drives (old for new), and start solr.
>> > (Solr
>> > > now looks for the index at /old/solr again, and finds it there.)
>> > > 7. Delete the index pointing to /new created in step 1.
>> > >
>> > > The idea was that this would create a new index for solr, would
>> populate
>> > it
>> > > with the right content, and would avoid having to touch our existing
>> solr
>> > > configurations aside from creating one new index, which we could soon
>> > > delete.
>> > >
>> > > I just did steps 1-5, but I got null pointer exceptions when starting
>> > solr,
>> > > and it appears that the index on /new has been almost completely
>> deleted
>> > by
>> > > Solr (this is a bummer, since it takes days to populate).
>> > >
>> > > Is this expected? Am I terribly crazy to try to swap indexes on disk?
>> As
>> > > far as I know, the only difference between the indexes is their name.
>> > >
>> > > We're using Solr version 4.10.4.
>> > >
>> > > Thank you,
>> > >
>> > > Mike
>> > >
>> >
>>
Re: Swapping indexes on disk
Posted by Mike Lissner <ml...@michaeljaylissner.com>.
I figured Solr would have a native system built in, but since we don't use
it already, I didn't want to learn all of its ins and outs just for this
disk situation.
Ditto, essentially, applies for the swapping strategy. We don't have a Solr
expert, just me, a generalist, and sorting out these kinds of things can
take a while. The hope was to avoid that kind of complication with some
clever use of symlinks and minor downtime. Our front end has a retry
mechanism, so if solr is down for less than a minute, users will just have
delayed responses, which is fine.
The new strategy is to rsync the files while solr is live, stop solr, do a
rsync diff, then start solr again. That'll give a bit for bit copy with
very little downtime — it's the strategy postgres recommends for disk-based
backups, so it seems like a safer bet. We needed a re-index anyway due to
schema changes, which my first attempt included, but I guess that'll have
to wait.
Thanks for the replies. If anybody can explain why the first strategy
failed, I'd still be interested in learning.
Mike
On Wed, Jun 14, 2017 at 12:09 PM Chris Ulicny <cu...@iq.media> wrote:
> Are you physically swapping the disks to introduce the new index? Or having
> both disks mounted at the same time?
>
> If the disks are simultaneously available, can you just swap the cores and
> then delete the core on the old disk?
>
> https://cwiki.apache.org/confluence/display/solr/CoreAdmin+API#CoreAdminAPI-SWAP
>
> We periodically move cores to different drives using solr's replication
> functionality and core swapping (after stopping replication). However, I've
> never encountered solr deleting an index like that.
>
>
>
> On Wed, Jun 14, 2017 at 2:48 PM David Hastings <
> hastings.recursive@gmail.com>
> wrote:
>
> > I dont have an answer to why the folder got cleared, however i am
> wondering
> > why you arent using basic replication to do this exact same thing, since
> > solr will natively take care of all this for you with no interruption to
> > the user and no stop/start routines etc.
> >
> > On Wed, Jun 14, 2017 at 2:26 PM, Mike Lissner <
> > mlissner@michaeljaylissner.com> wrote:
> >
> > > We are replacing a drive mounted at /old with one mounted at /new. Our
> > > index currently lives on /old, and our plan was to:
> > >
> > > 1. Create a new index on /new
> > > 2. Reindex from our database so that the new index on /new is properly
> > > populated.
> > > 3. Stop solr.
> > > 4. Symlink /old to /new (Solr now looks for the index at /old/solr,
> which
> > > redirects to /new/solr)
> > > 5. Start solr
> > > 6. (Later) Stop solr, swap the drives (old for new), and start solr.
> > (Solr
> > > now looks for the index at /old/solr again, and finds it there.)
> > > 7. Delete the index pointing to /new created in step 1.
> > >
> > > The idea was that this would create a new index for solr, would
> populate
> > it
> > > with the right content, and would avoid having to touch our existing
> solr
> > > configurations aside from creating one new index, which we could soon
> > > delete.
> > >
> > > I just did steps 1-5, but I got null pointer exceptions when starting
> > solr,
> > > and it appears that the index on /new has been almost completely
> deleted
> > by
> > > Solr (this is a bummer, since it takes days to populate).
> > >
> > > Is this expected? Am I terribly crazy to try to swap indexes on disk?
> As
> > > far as I know, the only difference between the indexes is their name.
> > >
> > > We're using Solr version 4.10.4.
> > >
> > > Thank you,
> > >
> > > Mike
> > >
> >
>
Re: Swapping indexes on disk
Posted by Chris Ulicny <cu...@iq.media>.
Are you physically swapping the disks to introduce the new index? Or having
both disks mounted at the same time?
If the disks are simultaneously available, can you just swap the cores and
then delete the core on the old disk?
https://cwiki.apache.org/confluence/display/solr/CoreAdmin+API#CoreAdminAPI-SWAP
We periodically move cores to different drives using solr's replication
functionality and core swapping (after stopping replication). However, I've
never encountered solr deleting an index like that.
On Wed, Jun 14, 2017 at 2:48 PM David Hastings <ha...@gmail.com>
wrote:
> I dont have an answer to why the folder got cleared, however i am wondering
> why you arent using basic replication to do this exact same thing, since
> solr will natively take care of all this for you with no interruption to
> the user and no stop/start routines etc.
>
> On Wed, Jun 14, 2017 at 2:26 PM, Mike Lissner <
> mlissner@michaeljaylissner.com> wrote:
>
> > We are replacing a drive mounted at /old with one mounted at /new. Our
> > index currently lives on /old, and our plan was to:
> >
> > 1. Create a new index on /new
> > 2. Reindex from our database so that the new index on /new is properly
> > populated.
> > 3. Stop solr.
> > 4. Symlink /old to /new (Solr now looks for the index at /old/solr, which
> > redirects to /new/solr)
> > 5. Start solr
> > 6. (Later) Stop solr, swap the drives (old for new), and start solr.
> (Solr
> > now looks for the index at /old/solr again, and finds it there.)
> > 7. Delete the index pointing to /new created in step 1.
> >
> > The idea was that this would create a new index for solr, would populate
> it
> > with the right content, and would avoid having to touch our existing solr
> > configurations aside from creating one new index, which we could soon
> > delete.
> >
> > I just did steps 1-5, but I got null pointer exceptions when starting
> solr,
> > and it appears that the index on /new has been almost completely deleted
> by
> > Solr (this is a bummer, since it takes days to populate).
> >
> > Is this expected? Am I terribly crazy to try to swap indexes on disk? As
> > far as I know, the only difference between the indexes is their name.
> >
> > We're using Solr version 4.10.4.
> >
> > Thank you,
> >
> > Mike
> >
>
Re: Swapping indexes on disk
Posted by David Hastings <ha...@gmail.com>.
I dont have an answer to why the folder got cleared, however i am wondering
why you arent using basic replication to do this exact same thing, since
solr will natively take care of all this for you with no interruption to
the user and no stop/start routines etc.
On Wed, Jun 14, 2017 at 2:26 PM, Mike Lissner <
mlissner@michaeljaylissner.com> wrote:
> We are replacing a drive mounted at /old with one mounted at /new. Our
> index currently lives on /old, and our plan was to:
>
> 1. Create a new index on /new
> 2. Reindex from our database so that the new index on /new is properly
> populated.
> 3. Stop solr.
> 4. Symlink /old to /new (Solr now looks for the index at /old/solr, which
> redirects to /new/solr)
> 5. Start solr
> 6. (Later) Stop solr, swap the drives (old for new), and start solr. (Solr
> now looks for the index at /old/solr again, and finds it there.)
> 7. Delete the index pointing to /new created in step 1.
>
> The idea was that this would create a new index for solr, would populate it
> with the right content, and would avoid having to touch our existing solr
> configurations aside from creating one new index, which we could soon
> delete.
>
> I just did steps 1-5, but I got null pointer exceptions when starting solr,
> and it appears that the index on /new has been almost completely deleted by
> Solr (this is a bummer, since it takes days to populate).
>
> Is this expected? Am I terribly crazy to try to swap indexes on disk? As
> far as I know, the only difference between the indexes is their name.
>
> We're using Solr version 4.10.4.
>
> Thank you,
>
> Mike
>