You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Furkan KAMACI <fu...@gmail.com> on 2013/04/25 23:22:52 UTC
How To Make Index Backup at SolrCloud?
I use SolrCloud. Let's assume that I want to move all indexes from one
place to another. There maybe two reasons for that:
First one is that: I will close all my system and I will use new machines
with previous indexes (if it is a must they may have same network topology)
at anywhere else after some time later.
Second one is that: I know that SolrCloud handles failures but I will back
up my indexes for a disaster event.
How can I back up my indexes? I know that I can start up new nodes and I
can close the old ones so I can move my indexes to other machines. However
how can I do such kind of backup (should I just copy data folder of Solr
nodes and put them to new Solr nodes after I change Zookeeper
configuration)?
What folks do?
Re: How To Make Index Backup at SolrCloud?
Posted by Timothy Potter <th...@gmail.com>.
Hi Furkan,
So here's what I do (not saying this is the best method, but
definitely works great albeit with a little work on my part)
The replication handler (which must be enabled for Solr cloud)
supports a backup command, e.g.
.../replication?command=backup&location=/mnt/backups
>From what I've heard, this is the "safe" way to backup an index in Solr.
Every X minutes, I send a hard commit and then send this "backup"
command to only 1 replica per shard as you don't need multiple
replicas of the same shard backing up the same thing. I chose to send
this to the leaders but that is definitely not required.
The actual backup runs in the background asynchronously, so the
command returns to your client immediately. So there's not a good way
to know when the backup is done ... so I "poll" the replication
handler's details action, e.g. .../replication?command=details, which
will return a completed on date when your backup is done. So my
backup tool simply waits until all shards report as being completed.
Then I move the backup offsite, which for us is S3.
Incidentally, I wrote my backup driver in Java b/c the SolrJ library
gives you access to all the cluster state information you need to pick
one replica per shard to start the backup on, e.g. something like
this:
private static final Map<String,String>
getShardLeaders(CloudSolrServer solr, String collection) throws
Exception {
Map<String,String> leaders = new TreeMap<String,String>();
ZkStateReader zkStateReader = solr.getZkStateReader();
for (Slice slice :
zkStateReader.getClusterState().getSlices(collection)) {
leaders.put(slice.getName(),
zkStateReader.getLeaderUrl(collection, slice.getName(), ZK_TIMEOUT));
}
return leaders;
}
On Thu, Apr 25, 2013 at 3:22 PM, Furkan KAMACI <fu...@gmail.com> wrote:
> I use SolrCloud. Let's assume that I want to move all indexes from one
> place to another. There maybe two reasons for that:
>
> First one is that: I will close all my system and I will use new machines
> with previous indexes (if it is a must they may have same network topology)
> at anywhere else after some time later.
> Second one is that: I know that SolrCloud handles failures but I will back
> up my indexes for a disaster event.
>
> How can I back up my indexes? I know that I can start up new nodes and I
> can close the old ones so I can move my indexes to other machines. However
> how can I do such kind of backup (should I just copy data folder of Solr
> nodes and put them to new Solr nodes after I change Zookeeper
> configuration)?
>
> What folks do?
Re: How To Make Index Backup at SolrCloud?
Posted by Upayavira <uv...@odoko.co.uk>.
>From looking at the ReplicationHandler code, it looks like if you hit it
with a 'details' request, it'll show you the details of the most recent
backup, including file count, status and completion time.
Upayavira
On Mon, May 20, 2013, at 08:46 AM, Furkan KAMACI wrote:
> Ooops, you didn't say it OK. It is at Timothy's answer.
>
> 2013/5/20 Otis Gospodnetic <ot...@gmail.com>
>
> > Hm, did I really say that? What was the context? Because I don't see
> > that in my response below....
> >
> > Otis
> > --
> > Search Analytics - http://sematext.com/search-analytics/index.html
> > SOLR Performance Monitoring - http://sematext.com/spm/index.html
> >
> >
> >
> >
> >
> > On Sun, May 19, 2013 at 10:16 AM, Furkan KAMACI <fu...@gmail.com>
> > wrote:
> > > Hi Otis;
> > >
> > > You said:
> > >
> > > "which will return a completed on date when your backup is done"
> > >
> > > which field is that?
> > >
> > > 2013/4/26 Otis Gospodnetic <ot...@gmail.com>
> > >
> > >> You can use the index backup command that's part of index replication,
> > >> check the Wiki.
> > >>
> > >> Otis
> > >> Solr & ElasticSearch Support
> > >> http://sematext.com/
> > >> On Apr 25, 2013 5:23 PM, "Furkan KAMACI" <fu...@gmail.com>
> > wrote:
> > >>
> > >> > I use SolrCloud. Let's assume that I want to move all indexes from one
> > >> > place to another. There maybe two reasons for that:
> > >> >
> > >> > First one is that: I will close all my system and I will use new
> > machines
> > >> > with previous indexes (if it is a must they may have same network
> > >> topology)
> > >> > at anywhere else after some time later.
> > >> > Second one is that: I know that SolrCloud handles failures but I will
> > >> back
> > >> > up my indexes for a disaster event.
> > >> >
> > >> > How can I back up my indexes? I know that I can start up new nodes
> > and I
> > >> > can close the old ones so I can move my indexes to other machines.
> > >> However
> > >> > how can I do such kind of backup (should I just copy data folder of
> > Solr
> > >> > nodes and put them to new Solr nodes after I change Zookeeper
> > >> > configuration)?
> > >> >
> > >> > What folks do?
> > >> >
> > >>
> >
Re: How To Make Index Backup at SolrCloud?
Posted by Furkan KAMACI <fu...@gmail.com>.
OK, I found it, no problem.
2013/5/20 Furkan KAMACI <fu...@gmail.com>
> Ooops, you didn't say it OK. It is at Timothy's answer.
>
>
> 2013/5/20 Otis Gospodnetic <ot...@gmail.com>
>
>> Hm, did I really say that? What was the context? Because I don't see
>> that in my response below....
>>
>> Otis
>> --
>> Search Analytics - http://sematext.com/search-analytics/index.html
>> SOLR Performance Monitoring - http://sematext.com/spm/index.html
>>
>>
>>
>>
>>
>> On Sun, May 19, 2013 at 10:16 AM, Furkan KAMACI <fu...@gmail.com>
>> wrote:
>> > Hi Otis;
>> >
>> > You said:
>> >
>> > "which will return a completed on date when your backup is done"
>> >
>> > which field is that?
>> >
>> > 2013/4/26 Otis Gospodnetic <ot...@gmail.com>
>> >
>> >> You can use the index backup command that's part of index replication,
>> >> check the Wiki.
>> >>
>> >> Otis
>> >> Solr & ElasticSearch Support
>> >> http://sematext.com/
>> >> On Apr 25, 2013 5:23 PM, "Furkan KAMACI" <fu...@gmail.com>
>> wrote:
>> >>
>> >> > I use SolrCloud. Let's assume that I want to move all indexes from
>> one
>> >> > place to another. There maybe two reasons for that:
>> >> >
>> >> > First one is that: I will close all my system and I will use new
>> machines
>> >> > with previous indexes (if it is a must they may have same network
>> >> topology)
>> >> > at anywhere else after some time later.
>> >> > Second one is that: I know that SolrCloud handles failures but I will
>> >> back
>> >> > up my indexes for a disaster event.
>> >> >
>> >> > How can I back up my indexes? I know that I can start up new nodes
>> and I
>> >> > can close the old ones so I can move my indexes to other machines.
>> >> However
>> >> > how can I do such kind of backup (should I just copy data folder of
>> Solr
>> >> > nodes and put them to new Solr nodes after I change Zookeeper
>> >> > configuration)?
>> >> >
>> >> > What folks do?
>> >> >
>> >>
>>
>
>
Re: How To Make Index Backup at SolrCloud?
Posted by Furkan KAMACI <fu...@gmail.com>.
Ooops, you didn't say it OK. It is at Timothy's answer.
2013/5/20 Otis Gospodnetic <ot...@gmail.com>
> Hm, did I really say that? What was the context? Because I don't see
> that in my response below....
>
> Otis
> --
> Search Analytics - http://sematext.com/search-analytics/index.html
> SOLR Performance Monitoring - http://sematext.com/spm/index.html
>
>
>
>
>
> On Sun, May 19, 2013 at 10:16 AM, Furkan KAMACI <fu...@gmail.com>
> wrote:
> > Hi Otis;
> >
> > You said:
> >
> > "which will return a completed on date when your backup is done"
> >
> > which field is that?
> >
> > 2013/4/26 Otis Gospodnetic <ot...@gmail.com>
> >
> >> You can use the index backup command that's part of index replication,
> >> check the Wiki.
> >>
> >> Otis
> >> Solr & ElasticSearch Support
> >> http://sematext.com/
> >> On Apr 25, 2013 5:23 PM, "Furkan KAMACI" <fu...@gmail.com>
> wrote:
> >>
> >> > I use SolrCloud. Let's assume that I want to move all indexes from one
> >> > place to another. There maybe two reasons for that:
> >> >
> >> > First one is that: I will close all my system and I will use new
> machines
> >> > with previous indexes (if it is a must they may have same network
> >> topology)
> >> > at anywhere else after some time later.
> >> > Second one is that: I know that SolrCloud handles failures but I will
> >> back
> >> > up my indexes for a disaster event.
> >> >
> >> > How can I back up my indexes? I know that I can start up new nodes
> and I
> >> > can close the old ones so I can move my indexes to other machines.
> >> However
> >> > how can I do such kind of backup (should I just copy data folder of
> Solr
> >> > nodes and put them to new Solr nodes after I change Zookeeper
> >> > configuration)?
> >> >
> >> > What folks do?
> >> >
> >>
>
Re: How To Make Index Backup at SolrCloud?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hm, did I really say that? What was the context? Because I don't see
that in my response below....
Otis
--
Search Analytics - http://sematext.com/search-analytics/index.html
SOLR Performance Monitoring - http://sematext.com/spm/index.html
On Sun, May 19, 2013 at 10:16 AM, Furkan KAMACI <fu...@gmail.com> wrote:
> Hi Otis;
>
> You said:
>
> "which will return a completed on date when your backup is done"
>
> which field is that?
>
> 2013/4/26 Otis Gospodnetic <ot...@gmail.com>
>
>> You can use the index backup command that's part of index replication,
>> check the Wiki.
>>
>> Otis
>> Solr & ElasticSearch Support
>> http://sematext.com/
>> On Apr 25, 2013 5:23 PM, "Furkan KAMACI" <fu...@gmail.com> wrote:
>>
>> > I use SolrCloud. Let's assume that I want to move all indexes from one
>> > place to another. There maybe two reasons for that:
>> >
>> > First one is that: I will close all my system and I will use new machines
>> > with previous indexes (if it is a must they may have same network
>> topology)
>> > at anywhere else after some time later.
>> > Second one is that: I know that SolrCloud handles failures but I will
>> back
>> > up my indexes for a disaster event.
>> >
>> > How can I back up my indexes? I know that I can start up new nodes and I
>> > can close the old ones so I can move my indexes to other machines.
>> However
>> > how can I do such kind of backup (should I just copy data folder of Solr
>> > nodes and put them to new Solr nodes after I change Zookeeper
>> > configuration)?
>> >
>> > What folks do?
>> >
>>
Re: How To Make Index Backup at SolrCloud?
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi Otis;
You said:
"which will return a completed on date when your backup is done"
which field is that?
2013/4/26 Otis Gospodnetic <ot...@gmail.com>
> You can use the index backup command that's part of index replication,
> check the Wiki.
>
> Otis
> Solr & ElasticSearch Support
> http://sematext.com/
> On Apr 25, 2013 5:23 PM, "Furkan KAMACI" <fu...@gmail.com> wrote:
>
> > I use SolrCloud. Let's assume that I want to move all indexes from one
> > place to another. There maybe two reasons for that:
> >
> > First one is that: I will close all my system and I will use new machines
> > with previous indexes (if it is a must they may have same network
> topology)
> > at anywhere else after some time later.
> > Second one is that: I know that SolrCloud handles failures but I will
> back
> > up my indexes for a disaster event.
> >
> > How can I back up my indexes? I know that I can start up new nodes and I
> > can close the old ones so I can move my indexes to other machines.
> However
> > how can I do such kind of backup (should I just copy data folder of Solr
> > nodes and put them to new Solr nodes after I change Zookeeper
> > configuration)?
> >
> > What folks do?
> >
>
Re: How To Make Index Backup at SolrCloud?
Posted by Otis Gospodnetic <ot...@gmail.com>.
You can use the index backup command that's part of index replication,
check the Wiki.
Otis
Solr & ElasticSearch Support
http://sematext.com/
On Apr 25, 2013 5:23 PM, "Furkan KAMACI" <fu...@gmail.com> wrote:
> I use SolrCloud. Let's assume that I want to move all indexes from one
> place to another. There maybe two reasons for that:
>
> First one is that: I will close all my system and I will use new machines
> with previous indexes (if it is a must they may have same network topology)
> at anywhere else after some time later.
> Second one is that: I know that SolrCloud handles failures but I will back
> up my indexes for a disaster event.
>
> How can I back up my indexes? I know that I can start up new nodes and I
> can close the old ones so I can move my indexes to other machines. However
> how can I do such kind of backup (should I just copy data folder of Solr
> nodes and put them to new Solr nodes after I change Zookeeper
> configuration)?
>
> What folks do?
>