You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sandeep Kumar Anumalla <sa...@etisalat.ae> on 2013/03/21 08:23:12 UTC
Solr index Backup and restore of large indexs
Hi,
We are loading daily 1TB (Apprx) of index data .Please let me know the best procedure to take Backup and restore of the indexes. I am using Solr 4.2.
Thanks & Regards
Sandeep A
Ext : 02618-2856
M : 0502493820
________________________________
The content of this email together with any attachments, statements and opinions expressed herein contains information that is private and confidential are intended for the named addressee(s) only. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. If you have received this message in error please notify postmaster@etisalat.ae by email immediately and delete the message without making any copies.
RE: Solr index Backup and restore of large indexs
Posted by Sandeep Kumar Anumalla <sa...@etisalat.ae>.
Hi,
I am exploring all the possible options.
We want to distribute 1 TB traffic among 3 Slor Shards(Masters) and corresponding 3 Solr Slaves.
Initially I have used Master/Slave setup. But in this case my traffic rate on Master is very high, because of this we are facing the blow issue while replicating to Slave.
-------------------------------------
SnapPull failed
SEVERE: SnapPull failed :org.apache.solr.common.SolrException: Unable to download _xv0_Lucene41_0.doc completely. Downloaded 0!=5935
------------------------------------
In this case the Slave machine also has to be the same Hardware and Software configuration as such the Master; this seems to be more expensive.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Then I decided to use multiple Solr instances on single machine and accessing them by Using "EmbeddedSolrServer", and planned to query all these instances to get the required result.
In this case there is no need of Slave machine,just we need to take the backup and we can store in any external hard disks.
Here there are 2 issues I am facing.
1. Loading is not that much fast when compare to Database.
2. How to take incremental backup? Means I don't want to take the full back up every time.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Thanks
Sandeep A
-----Original Message-----
From: Joel Bernstein [mailto:joelsolr@gmail.com]
Sent: 28 March, 2013 04:51 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr index Backup and restore of large indexs
Hi,
Are you running Solr Cloud or Master/Slave? I'm assuming with 1TB a day you're sharding.
With master/slave you can configure incremental index replication to another core. The backup core can be local on the server, on a separate sever or in a separate data center.
With Solr Cloud replicas can be setup to automatically have redundant copies of the index. These copies though are live copies and will handle queries. Replicating data to a separate data center is typically not done through Solr Cloud replication.
Joel
On Mon, Mar 25, 2013 at 11:43 PM, Otis Gospodnetic < otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> Try something like this: http://host/solr/replication?command=backup
>
> See: http://wiki.apache.org/solr/SolrReplication
>
> Otis
> --
> Solr & ElasticSearch Support
> http://sematext.com/
>
>
>
>
>
> On Thu, Mar 21, 2013 at 3:23 AM, Sandeep Kumar Anumalla
> <sa...@etisalat.ae> wrote:
> >
> > Hi,
> >
> > We are loading daily 1TB (Apprx) of index data .Please let me know
> > the
> best procedure to take Backup and restore of the indexes. I am using
> Solr 4.2.
> >
> >
> >
> > Thanks & Regards
> > Sandeep A
> > Ext : 02618-2856
> > M : 0502493820
> >
> >
> > ________________________________
> > The content of this email together with any attachments, statements
> > and
> opinions expressed herein contains information that is private and
> confidential are intended for the named addressee(s) only. If you are
> not the addressee of this email you may not copy, forward, disclose or
> otherwise use it or any part of it in any form whatsoever. If you have
> received this message in error please notify postmaster@etisalat.ae by
> email immediately and delete the message without making any copies.
>
--
Joel Bernstein
Professional Services LucidWorks
The content of this email together with any attachments, statements and opinions expressed herein contains information that is private and confidential are intended for the named addressee(s) only. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. If you have received this message in error please notify postmaster@etisalat.ae by email immediately and delete the message without making any copies.
RE: Solr index Backup and restore of large indexs
Posted by Sandeep Kumar Anumalla <sa...@etisalat.ae>.
Please update?
-----Original Message-----
From: Sandeep Kumar Anumalla
Sent: 31 March, 2013 12:08 PM
To: solr-user@lucene.apache.org
Cc: 'Joel Bernstein'
Subject: RE: Solr index Backup and restore of large indexs
Hi,
I am exploring all the possible options.
We want to distribute 1 TB traffic among 3 Slor Shards(Masters) and corresponding 3 Solr Slaves.
Initially I have used Master/Slave setup. But in this case my traffic rate on Master is very high, because of this we are facing the blow issue while replicating to Slave.
-------------------------------------
SnapPull failed
SEVERE: SnapPull failed :org.apache.solr.common.SolrException: Unable to download _xv0_Lucene41_0.doc completely. Downloaded 0!=5935
------------------------------------
In this case the Slave machine also has to be the same Hardware and Software configuration as such the Master; this seems to be more expensive.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Then I decided to use multiple Solr instances on single machine and accessing them by Using "EmbeddedSolrServer", and planned to query all these instances to get the required result.
In this case there is no need of Slave machine,just we need to take the backup and we can store in any external hard disks.
Here there are 2 issues I am facing.
1. Loading is not that much fast when compare to Database.
2. How to take incremental backup? Means I don't want to take the full back up every time.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Thanks
Sandeep A
-----Original Message-----
From: Joel Bernstein [mailto:joelsolr@gmail.com]
Sent: 28 March, 2013 04:51 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr index Backup and restore of large indexs
Hi,
Are you running Solr Cloud or Master/Slave? I'm assuming with 1TB a day you're sharding.
With master/slave you can configure incremental index replication to another core. The backup core can be local on the server, on a separate sever or in a separate data center.
With Solr Cloud replicas can be setup to automatically have redundant copies of the index. These copies though are live copies and will handle queries. Replicating data to a separate data center is typically not done through Solr Cloud replication.
Joel
On Mon, Mar 25, 2013 at 11:43 PM, Otis Gospodnetic < otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> Try something like this: http://host/solr/replication?command=backup
>
> See: http://wiki.apache.org/solr/SolrReplication
>
> Otis
> --
> Solr & ElasticSearch Support
> http://sematext.com/
>
>
>
>
>
> On Thu, Mar 21, 2013 at 3:23 AM, Sandeep Kumar Anumalla
> <sa...@etisalat.ae> wrote:
> >
> > Hi,
> >
> > We are loading daily 1TB (Apprx) of index data .Please let me know
> > the
> best procedure to take Backup and restore of the indexes. I am using
> Solr 4.2.
> >
> >
> >
> > Thanks & Regards
> > Sandeep A
> > Ext : 02618-2856
> > M : 0502493820
> >
> >
> > ________________________________
> > The content of this email together with any attachments, statements
> > and
> opinions expressed herein contains information that is private and
> confidential are intended for the named addressee(s) only. If you are
> not the addressee of this email you may not copy, forward, disclose or
> otherwise use it or any part of it in any form whatsoever. If you have
> received this message in error please notify postmaster@etisalat.ae by
> email immediately and delete the message without making any copies.
>
--
Joel Bernstein
Professional Services LucidWorks
The content of this email together with any attachments, statements and opinions expressed herein contains information that is private and confidential are intended for the named addressee(s) only. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. If you have received this message in error please notify postmaster@etisalat.ae by email immediately and delete the message without making any copies.
Re: Solr index Backup and restore of large indexs
Posted by Joel Bernstein <jo...@gmail.com>.
Hi,
Are you running Solr Cloud or Master/Slave? I'm assuming with 1TB a day
you're sharding.
With master/slave you can configure incremental index replication to
another core. The backup core can be local on the server, on a separate
sever or in a separate data center.
With Solr Cloud replicas can be setup to automatically have redundant
copies of the index. These copies though are live copies and will handle
queries. Replicating data to a separate data center is typically not done
through Solr Cloud replication.
Joel
On Mon, Mar 25, 2013 at 11:43 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> Try something like this: http://host/solr/replication?command=backup
>
> See: http://wiki.apache.org/solr/SolrReplication
>
> Otis
> --
> Solr & ElasticSearch Support
> http://sematext.com/
>
>
>
>
>
> On Thu, Mar 21, 2013 at 3:23 AM, Sandeep Kumar Anumalla
> <sa...@etisalat.ae> wrote:
> >
> > Hi,
> >
> > We are loading daily 1TB (Apprx) of index data .Please let me know the
> best procedure to take Backup and restore of the indexes. I am using Solr
> 4.2.
> >
> >
> >
> > Thanks & Regards
> > Sandeep A
> > Ext : 02618-2856
> > M : 0502493820
> >
> >
> > ________________________________
> > The content of this email together with any attachments, statements and
> opinions expressed herein contains information that is private and
> confidential are intended for the named addressee(s) only. If you are not
> the addressee of this email you may not copy, forward, disclose or
> otherwise use it or any part of it in any form whatsoever. If you have
> received this message in error please notify postmaster@etisalat.ae by
> email immediately and delete the message without making any copies.
>
--
Joel Bernstein
Professional Services LucidWorks
Re: Solr index Backup and restore of large indexs
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
Try something like this: http://host/solr/replication?command=backup
See: http://wiki.apache.org/solr/SolrReplication
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Thu, Mar 21, 2013 at 3:23 AM, Sandeep Kumar Anumalla
<sa...@etisalat.ae> wrote:
>
> Hi,
>
> We are loading daily 1TB (Apprx) of index data .Please let me know the best procedure to take Backup and restore of the indexes. I am using Solr 4.2.
>
>
>
> Thanks & Regards
> Sandeep A
> Ext : 02618-2856
> M : 0502493820
>
>
> ________________________________
> The content of this email together with any attachments, statements and opinions expressed herein contains information that is private and confidential are intended for the named addressee(s) only. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. If you have received this message in error please notify postmaster@etisalat.ae by email immediately and delete the message without making any copies.