You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bram Van Dam <br...@intix.eu> on 2020/08/06 14:18:05 UTC
Backups in SolrCloud using snapshots of individual cores?
Hey folks,
Been reading up about the various ways of creating backups. The whole
"shared filesystem for Solrcloud backups"-thing is kind of a no-go in
our environment, so I've been looking for ways around that, and here's
what I've come up with so far:
1. Stop applications from writing to solr
2. Commit everything
3. Identify a single core for each shard in each collection
4. Snapshot that core using CREATESNAPSHOT in the Collections API
5. Once complete, re-enable application write access to Solr
6. Create a backup from these snapshots using the replication handler's
backup function (replication?command=backup&commitName=mySnapshot)
7. Put the backups somewhere safe
8. Clean up snapshots
This seems ... too good to be true? I've seen so many threads about how
hard it is to create backups in SolrCloud on this mailing list over the
years, but this seems pretty straightforward? Am I missing some
glaringly obvious reason why this will fail catastrophically?
Using Solr 7.7 in this case.
Feedback much appreciated!
Thanks,
- Bram
Re: Backups in SolrCloud using snapshots of individual cores?
Posted by Bram Van Dam <br...@intix.eu>.
On 11/08/2020 13:15, Erick Erickson wrote:
> CDCR is being deprecated. so I wouldn’t suggest it for the long term.
Ah yes, thanks for pointing that out. That makes Dominique's alternative
less attractive. I guess I'll stick to my original proposal!
Thanks Erick :-)
- Bram
Re: Backups in SolrCloud using snapshots of individual cores?
Posted by Dominique Bejean <do...@eolya.fr>.
An idea could be use autoscaling API in order to add a PULL replica for
each shard located in one or more low resource backup dedicated nodes in
separate hardware.
However, we need to exclude these "PULL backup replica" from searches.
Unfortunately, I am not aware of this possibility.
For better RPO, TLOG replica would be better, but it could become an NRT
replica.
So, may be one solution could be create a new BACKUP replica type with
these characteristics :
- According to RPO, options at creation time : based on PULL or TLOG
sync mode
- Search disabled
Dominique
Le mar. 11 août 2020 à 14:07, Erick Erickson <er...@gmail.com> a
écrit :
> Dominique:
>
> Alternatives are under discussion, there isn’t a recommendation yet.
>
> Erick
>
> > On Aug 11, 2020, at 7:49 AM, Dominique Bejean <do...@eolya.fr>
> wrote:
> >
> > I missed that !
> > Are you aware about an alternative ?
> >
> > Regards
> >
> > Dominique
> >
> >
> > Le mar. 11 août 2020 à 13:15, Erick Erickson <er...@gmail.com> a
> > écrit :
> >
> >> CDCR is being deprecated. so I wouldn’t suggest it for the long term.
> >>
> >>> On Aug 10, 2020, at 9:33 PM, Ashwin Ramesh <as...@canva.com.INVALID>
> >> wrote:
> >>>
> >>> I would love an answer to this too!
> >>>
> >>> On Fri, Aug 7, 2020 at 12:18 AM Bram Van Dam <br...@intix.eu>
> >> wrote:
> >>>
> >>>> Hey folks,
> >>>>
> >>>> Been reading up about the various ways of creating backups. The whole
> >>>> "shared filesystem for Solrcloud backups"-thing is kind of a no-go in
> >>>> our environment, so I've been looking for ways around that, and here's
> >>>> what I've come up with so far:
> >>>>
> >>>> 1. Stop applications from writing to solr
> >>>>
> >>>> 2. Commit everything
> >>>>
> >>>> 3. Identify a single core for each shard in each collection
> >>>>
> >>>> 4. Snapshot that core using CREATESNAPSHOT in the Collections API
> >>>>
> >>>> 5. Once complete, re-enable application write access to Solr
> >>>>
> >>>> 6. Create a backup from these snapshots using the replication
> handler's
> >>>> backup function (replication?command=backup&commitName=mySnapshot)
> >>>>
> >>>> 7. Put the backups somewhere safe
> >>>>
> >>>> 8. Clean up snapshots
> >>>>
> >>>>
> >>>> This seems ... too good to be true? I've seen so many threads about
> how
> >>>> hard it is to create backups in SolrCloud on this mailing list over
> the
> >>>> years, but this seems pretty straightforward? Am I missing some
> >>>> glaringly obvious reason why this will fail catastrophically?
> >>>>
> >>>> Using Solr 7.7 in this case.
> >>>>
> >>>> Feedback much appreciated!
> >>>>
> >>>> Thanks,
> >>>>
> >>>> - Bram
> >>>>
> >>>
> >>> --
> >>> **
> >>> ** <https://www.canva.com/>Empowering the world to design
> >>> Share accurate
> >>> information on COVID-19 and spread messages of support to your
> community.
> >>>
> >>> Here are some resources
> >>> <
> >>
> https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates
> >
> >>
> >>> that can help.
> >>> <https://twitter.com/canva> <https://facebook.com/canva>
> >>> <https://au.linkedin.com/company/canva> <https://twitter.com/canva>
> >>> <https://facebook.com/canva> <https://au.linkedin.com/company/canva>
> >>> <https://instagram.com/canva>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
>
>
Re: Backups in SolrCloud using snapshots of individual cores?
Posted by Erick Erickson <er...@gmail.com>.
Dominique:
Alternatives are under discussion, there isn’t a recommendation yet.
Erick
> On Aug 11, 2020, at 7:49 AM, Dominique Bejean <do...@eolya.fr> wrote:
>
> I missed that !
> Are you aware about an alternative ?
>
> Regards
>
> Dominique
>
>
> Le mar. 11 août 2020 à 13:15, Erick Erickson <er...@gmail.com> a
> écrit :
>
>> CDCR is being deprecated. so I wouldn’t suggest it for the long term.
>>
>>> On Aug 10, 2020, at 9:33 PM, Ashwin Ramesh <as...@canva.com.INVALID>
>> wrote:
>>>
>>> I would love an answer to this too!
>>>
>>> On Fri, Aug 7, 2020 at 12:18 AM Bram Van Dam <br...@intix.eu>
>> wrote:
>>>
>>>> Hey folks,
>>>>
>>>> Been reading up about the various ways of creating backups. The whole
>>>> "shared filesystem for Solrcloud backups"-thing is kind of a no-go in
>>>> our environment, so I've been looking for ways around that, and here's
>>>> what I've come up with so far:
>>>>
>>>> 1. Stop applications from writing to solr
>>>>
>>>> 2. Commit everything
>>>>
>>>> 3. Identify a single core for each shard in each collection
>>>>
>>>> 4. Snapshot that core using CREATESNAPSHOT in the Collections API
>>>>
>>>> 5. Once complete, re-enable application write access to Solr
>>>>
>>>> 6. Create a backup from these snapshots using the replication handler's
>>>> backup function (replication?command=backup&commitName=mySnapshot)
>>>>
>>>> 7. Put the backups somewhere safe
>>>>
>>>> 8. Clean up snapshots
>>>>
>>>>
>>>> This seems ... too good to be true? I've seen so many threads about how
>>>> hard it is to create backups in SolrCloud on this mailing list over the
>>>> years, but this seems pretty straightforward? Am I missing some
>>>> glaringly obvious reason why this will fail catastrophically?
>>>>
>>>> Using Solr 7.7 in this case.
>>>>
>>>> Feedback much appreciated!
>>>>
>>>> Thanks,
>>>>
>>>> - Bram
>>>>
>>>
>>> --
>>> **
>>> ** <https://www.canva.com/>Empowering the world to design
>>> Share accurate
>>> information on COVID-19 and spread messages of support to your community.
>>>
>>> Here are some resources
>>> <
>> https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates>
>>
>>> that can help.
>>> <https://twitter.com/canva> <https://facebook.com/canva>
>>> <https://au.linkedin.com/company/canva> <https://twitter.com/canva>
>>> <https://facebook.com/canva> <https://au.linkedin.com/company/canva>
>>> <https://instagram.com/canva>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
Re: Backups in SolrCloud using snapshots of individual cores?
Posted by Dominique Bejean <do...@eolya.fr>.
I missed that !
Are you aware about an alternative ?
Regards
Dominique
Le mar. 11 août 2020 à 13:15, Erick Erickson <er...@gmail.com> a
écrit :
> CDCR is being deprecated. so I wouldn’t suggest it for the long term.
>
> > On Aug 10, 2020, at 9:33 PM, Ashwin Ramesh <as...@canva.com.INVALID>
> wrote:
> >
> > I would love an answer to this too!
> >
> > On Fri, Aug 7, 2020 at 12:18 AM Bram Van Dam <br...@intix.eu>
> wrote:
> >
> >> Hey folks,
> >>
> >> Been reading up about the various ways of creating backups. The whole
> >> "shared filesystem for Solrcloud backups"-thing is kind of a no-go in
> >> our environment, so I've been looking for ways around that, and here's
> >> what I've come up with so far:
> >>
> >> 1. Stop applications from writing to solr
> >>
> >> 2. Commit everything
> >>
> >> 3. Identify a single core for each shard in each collection
> >>
> >> 4. Snapshot that core using CREATESNAPSHOT in the Collections API
> >>
> >> 5. Once complete, re-enable application write access to Solr
> >>
> >> 6. Create a backup from these snapshots using the replication handler's
> >> backup function (replication?command=backup&commitName=mySnapshot)
> >>
> >> 7. Put the backups somewhere safe
> >>
> >> 8. Clean up snapshots
> >>
> >>
> >> This seems ... too good to be true? I've seen so many threads about how
> >> hard it is to create backups in SolrCloud on this mailing list over the
> >> years, but this seems pretty straightforward? Am I missing some
> >> glaringly obvious reason why this will fail catastrophically?
> >>
> >> Using Solr 7.7 in this case.
> >>
> >> Feedback much appreciated!
> >>
> >> Thanks,
> >>
> >> - Bram
> >>
> >
> > --
> > **
> > ** <https://www.canva.com/>Empowering the world to design
> > Share accurate
> > information on COVID-19 and spread messages of support to your community.
> >
> > Here are some resources
> > <
> https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates>
>
> > that can help.
> > <https://twitter.com/canva> <https://facebook.com/canva>
> > <https://au.linkedin.com/company/canva> <https://twitter.com/canva>
> > <https://facebook.com/canva> <https://au.linkedin.com/company/canva>
> > <https://instagram.com/canva>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
Re: Backups in SolrCloud using snapshots of individual cores?
Posted by Erick Erickson <er...@gmail.com>.
CDCR is being deprecated. so I wouldn’t suggest it for the long term.
> On Aug 10, 2020, at 9:33 PM, Ashwin Ramesh <as...@canva.com.INVALID> wrote:
>
> I would love an answer to this too!
>
> On Fri, Aug 7, 2020 at 12:18 AM Bram Van Dam <br...@intix.eu> wrote:
>
>> Hey folks,
>>
>> Been reading up about the various ways of creating backups. The whole
>> "shared filesystem for Solrcloud backups"-thing is kind of a no-go in
>> our environment, so I've been looking for ways around that, and here's
>> what I've come up with so far:
>>
>> 1. Stop applications from writing to solr
>>
>> 2. Commit everything
>>
>> 3. Identify a single core for each shard in each collection
>>
>> 4. Snapshot that core using CREATESNAPSHOT in the Collections API
>>
>> 5. Once complete, re-enable application write access to Solr
>>
>> 6. Create a backup from these snapshots using the replication handler's
>> backup function (replication?command=backup&commitName=mySnapshot)
>>
>> 7. Put the backups somewhere safe
>>
>> 8. Clean up snapshots
>>
>>
>> This seems ... too good to be true? I've seen so many threads about how
>> hard it is to create backups in SolrCloud on this mailing list over the
>> years, but this seems pretty straightforward? Am I missing some
>> glaringly obvious reason why this will fail catastrophically?
>>
>> Using Solr 7.7 in this case.
>>
>> Feedback much appreciated!
>>
>> Thanks,
>>
>> - Bram
>>
>
> --
> **
> ** <https://www.canva.com/>Empowering the world to design
> Share accurate
> information on COVID-19 and spread messages of support to your community.
>
> Here are some resources
> <https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates>
> that can help.
> <https://twitter.com/canva> <https://facebook.com/canva>
> <https://au.linkedin.com/company/canva> <https://twitter.com/canva>
> <https://facebook.com/canva> <https://au.linkedin.com/company/canva>
> <https://instagram.com/canva>
>
>
>
>
>
>
>
>
>
>
Re: Backups in SolrCloud using snapshots of individual cores?
Posted by Ashwin Ramesh <as...@canva.com.INVALID>.
I would love an answer to this too!
On Fri, Aug 7, 2020 at 12:18 AM Bram Van Dam <br...@intix.eu> wrote:
> Hey folks,
>
> Been reading up about the various ways of creating backups. The whole
> "shared filesystem for Solrcloud backups"-thing is kind of a no-go in
> our environment, so I've been looking for ways around that, and here's
> what I've come up with so far:
>
> 1. Stop applications from writing to solr
>
> 2. Commit everything
>
> 3. Identify a single core for each shard in each collection
>
> 4. Snapshot that core using CREATESNAPSHOT in the Collections API
>
> 5. Once complete, re-enable application write access to Solr
>
> 6. Create a backup from these snapshots using the replication handler's
> backup function (replication?command=backup&commitName=mySnapshot)
>
> 7. Put the backups somewhere safe
>
> 8. Clean up snapshots
>
>
> This seems ... too good to be true? I've seen so many threads about how
> hard it is to create backups in SolrCloud on this mailing list over the
> years, but this seems pretty straightforward? Am I missing some
> glaringly obvious reason why this will fail catastrophically?
>
> Using Solr 7.7 in this case.
>
> Feedback much appreciated!
>
> Thanks,
>
> - Bram
>
--
**
** <https://www.canva.com/>Empowering the world to design
Share accurate
information on COVID-19 and spread messages of support to your community.
Here are some resources
<https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates>
that can help.
<https://twitter.com/canva> <https://facebook.com/canva>
<https://au.linkedin.com/company/canva> <https://twitter.com/canva>
<https://facebook.com/canva> <https://au.linkedin.com/company/canva>
<https://instagram.com/canva>
Re: Backups in SolrCloud using snapshots of individual cores?
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
This procedure looks fine but it is a little complexe to automatize.
Why not consider backup based on CDCR for Solrcloud or Replication for Solr
standalone ?
For Solrcloud, CDCR can be configured with source and target collections in
the same Solrcloud cluster. The target collection can have their shards
located in dedicated nodes and replication factor set to 1.
You need to be careful of locating target nodes on separate hardware (VM
and storage) and ideally in separate geographical locations.
You will be able to achieve very good RPO and RTO.
If RTO is not high, the dedicated nodes for backup destination can have few
CPU and RAM
If RTO is high we can imagine the backup becomes the live collection very
fast instead of restore or in degraded search only mode during restore.
Regards.
Dominique
Le jeu. 6 août 2020 à 16:18, Bram Van Dam <br...@intix.eu> a écrit :
> Hey folks,
>
> Been reading up about the various ways of creating backups. The whole
> "shared filesystem for Solrcloud backups"-thing is kind of a no-go in
> our environment, so I've been looking for ways around that, and here's
> what I've come up with so far:
>
> 1. Stop applications from writing to solr
>
> 2. Commit everything
>
> 3. Identify a single core for each shard in each collection
>
> 4. Snapshot that core using CREATESNAPSHOT in the Collections API
>
> 5. Once complete, re-enable application write access to Solr
>
> 6. Create a backup from these snapshots using the replication handler's
> backup function (replication?command=backup&commitName=mySnapshot)
>
> 7. Put the backups somewhere safe
>
> 8. Clean up snapshots
>
>
> This seems ... too good to be true? I've seen so many threads about how
> hard it is to create backups in SolrCloud on this mailing list over the
> years, but this seems pretty straightforward? Am I missing some
> glaringly obvious reason why this will fail catastrophically?
>
> Using Solr 7.7 in this case.
>
> Feedback much appreciated!
>
> Thanks,
>
> - Bram
>