You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ashwin Ramesh <as...@canva.com.INVALID> on 2020/03/10 06:53:33 UTC
Overseer & Backups - Questions
Hi everybody,
Quick Specs:
- Solr 7.4 Solr Cloud
- 30gb index on 8 shards Tlog/Pull
We run daily backups on our 30gb index and noticed that the overseer does
not process other jobs on it's task list while the backup is being taken.
They remain on the pending list (in ZK). Is this expected?
Also I was wondering if there was a safe way to cancel a currently running
task or deleing pending tasks?
Regards,
Ash
--
**
** <https://www.canva.com/>Empowering the world to design
Also, we're
hiring. Apply here! <https://about.canva.com/careers/>
<https://twitter.com/canva> <https://facebook.com/canva>
<https://au.linkedin.com/company/canva> <https://twitter.com/canva>
<https://facebook.com/canva> <https://au.linkedin.com/company/canva>
<https://instagram.com/canva>
Re: Overseer & Backups - Questions
Posted by Ashwin Ramesh <as...@canva.com.INVALID>.
Hey Aroop,
Yes we sent ASYNC=<SOME_UUID>
Backups are taken to an EFS drive (AWS's managed NFS)
I also thought it was async and Solr can process multiple tasks at once.
But the ZK state definitely showed that only the backup task was in
progress while all the other tasks were queued up.
Regards,
Ash
On Wed, Mar 11, 2020 at 9:21 AM Aroop Ganguly
<ar...@icloud.com.invalid> wrote:
> Backups on hdfs ?
> These should not be blocking if invoked asynchronously, are u doing them
> async by passing the async flag?
>
> > On Mar 10, 2020, at 3:19 PM, Ashwin Ramesh <as...@canva.com.INVALID>
> wrote:
> >
> > We use the collection API to invoke backups. The tasks we noticed that
> > stalled are ADDREPLICA. As expected when the backup completed a few hours
> > ago, the task then got completed. Is there some concurrency setting with
> > these tasks? Or is a backup a blocking task? We noticed that the index
> was
> > still being flushed to segments though.
> >
> > Regards,
> >
> > Ash
> >
> > On Wed, Mar 11, 2020 at 3:18 AM Aroop Ganguly
> > <ar...@icloud.com.invalid> wrote:
> >
> >> May we know how you are invoking backups ?
> >>
> >>> On Mar 9, 2020, at 11:53 PM, Ashwin Ramesh <as...@canva.com.INVALID>
> >> wrote:
> >>>
> >>> Hi everybody,
> >>>
> >>> Quick Specs:
> >>> - Solr 7.4 Solr Cloud
> >>> - 30gb index on 8 shards Tlog/Pull
> >>>
> >>> We run daily backups on our 30gb index and noticed that the overseer
> does
> >>> not process other jobs on it's task list while the backup is being
> taken.
> >>> They remain on the pending list (in ZK). Is this expected?
> >>>
> >>> Also I was wondering if there was a safe way to cancel a currently
> >> running
> >>> task or deleing pending tasks?
> >>>
> >>> Regards,
> >>>
> >>> Ash
> >>>
> >>> --
> >>> **
> >>> ** <https://www.canva.com/>Empowering the world to design
> >>> Also, we're
> >>> hiring. Apply here! <https://about.canva.com/careers/>
> >>>
> >>> <https://twitter.com/canva> <https://facebook.com/canva>
> >>> <https://au.linkedin.com/company/canva> <https://twitter.com/canva>
> >>> <https://facebook.com/canva> <https://au.linkedin.com/company/canva>
> >>> <https://instagram.com/canva>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >
> > --
> > **
> > ** <https://www.canva.com/>Empowering the world to design
> > Also, we're
> > hiring. Apply here! <https://about.canva.com/careers/>
> >
> > <https://twitter.com/canva> <https://facebook.com/canva>
> > <https://au.linkedin.com/company/canva> <https://twitter.com/canva>
> > <https://facebook.com/canva> <https://au.linkedin.com/company/canva>
> > <https://instagram.com/canva>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
--
**
** <https://www.canva.com/>Empowering the world to design
Also, we're
hiring. Apply here! <https://about.canva.com/careers/>
<https://twitter.com/canva> <https://facebook.com/canva>
<https://au.linkedin.com/company/canva> <https://twitter.com/canva>
<https://facebook.com/canva> <https://au.linkedin.com/company/canva>
<https://instagram.com/canva>
Re: Overseer & Backups - Questions
Posted by Aroop Ganguly <ar...@icloud.com.INVALID>.
Backups on hdfs ?
These should not be blocking if invoked asynchronously, are u doing them async by passing the async flag?
> On Mar 10, 2020, at 3:19 PM, Ashwin Ramesh <as...@canva.com.INVALID> wrote:
>
> We use the collection API to invoke backups. The tasks we noticed that
> stalled are ADDREPLICA. As expected when the backup completed a few hours
> ago, the task then got completed. Is there some concurrency setting with
> these tasks? Or is a backup a blocking task? We noticed that the index was
> still being flushed to segments though.
>
> Regards,
>
> Ash
>
> On Wed, Mar 11, 2020 at 3:18 AM Aroop Ganguly
> <ar...@icloud.com.invalid> wrote:
>
>> May we know how you are invoking backups ?
>>
>>> On Mar 9, 2020, at 11:53 PM, Ashwin Ramesh <as...@canva.com.INVALID>
>> wrote:
>>>
>>> Hi everybody,
>>>
>>> Quick Specs:
>>> - Solr 7.4 Solr Cloud
>>> - 30gb index on 8 shards Tlog/Pull
>>>
>>> We run daily backups on our 30gb index and noticed that the overseer does
>>> not process other jobs on it's task list while the backup is being taken.
>>> They remain on the pending list (in ZK). Is this expected?
>>>
>>> Also I was wondering if there was a safe way to cancel a currently
>> running
>>> task or deleing pending tasks?
>>>
>>> Regards,
>>>
>>> Ash
>>>
>>> --
>>> **
>>> ** <https://www.canva.com/>Empowering the world to design
>>> Also, we're
>>> hiring. Apply here! <https://about.canva.com/careers/>
>>>
>>> <https://twitter.com/canva> <https://facebook.com/canva>
>>> <https://au.linkedin.com/company/canva> <https://twitter.com/canva>
>>> <https://facebook.com/canva> <https://au.linkedin.com/company/canva>
>>> <https://instagram.com/canva>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
> --
> **
> ** <https://www.canva.com/>Empowering the world to design
> Also, we're
> hiring. Apply here! <https://about.canva.com/careers/>
>
> <https://twitter.com/canva> <https://facebook.com/canva>
> <https://au.linkedin.com/company/canva> <https://twitter.com/canva>
> <https://facebook.com/canva> <https://au.linkedin.com/company/canva>
> <https://instagram.com/canva>
>
>
>
>
>
>
>
>
>
>
Re: Overseer & Backups - Questions
Posted by Ashwin Ramesh <as...@canva.com.INVALID>.
We use the collection API to invoke backups. The tasks we noticed that
stalled are ADDREPLICA. As expected when the backup completed a few hours
ago, the task then got completed. Is there some concurrency setting with
these tasks? Or is a backup a blocking task? We noticed that the index was
still being flushed to segments though.
Regards,
Ash
On Wed, Mar 11, 2020 at 3:18 AM Aroop Ganguly
<ar...@icloud.com.invalid> wrote:
> May we know how you are invoking backups ?
>
> > On Mar 9, 2020, at 11:53 PM, Ashwin Ramesh <as...@canva.com.INVALID>
> wrote:
> >
> > Hi everybody,
> >
> > Quick Specs:
> > - Solr 7.4 Solr Cloud
> > - 30gb index on 8 shards Tlog/Pull
> >
> > We run daily backups on our 30gb index and noticed that the overseer does
> > not process other jobs on it's task list while the backup is being taken.
> > They remain on the pending list (in ZK). Is this expected?
> >
> > Also I was wondering if there was a safe way to cancel a currently
> running
> > task or deleing pending tasks?
> >
> > Regards,
> >
> > Ash
> >
> > --
> > **
> > ** <https://www.canva.com/>Empowering the world to design
> > Also, we're
> > hiring. Apply here! <https://about.canva.com/careers/>
> >
> > <https://twitter.com/canva> <https://facebook.com/canva>
> > <https://au.linkedin.com/company/canva> <https://twitter.com/canva>
> > <https://facebook.com/canva> <https://au.linkedin.com/company/canva>
> > <https://instagram.com/canva>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
--
**
** <https://www.canva.com/>Empowering the world to design
Also, we're
hiring. Apply here! <https://about.canva.com/careers/>
<https://twitter.com/canva> <https://facebook.com/canva>
<https://au.linkedin.com/company/canva> <https://twitter.com/canva>
<https://facebook.com/canva> <https://au.linkedin.com/company/canva>
<https://instagram.com/canva>
Re: Overseer & Backups - Questions
Posted by Aroop Ganguly <ar...@icloud.com.INVALID>.
May we know how you are invoking backups ?
> On Mar 9, 2020, at 11:53 PM, Ashwin Ramesh <as...@canva.com.INVALID> wrote:
>
> Hi everybody,
>
> Quick Specs:
> - Solr 7.4 Solr Cloud
> - 30gb index on 8 shards Tlog/Pull
>
> We run daily backups on our 30gb index and noticed that the overseer does
> not process other jobs on it's task list while the backup is being taken.
> They remain on the pending list (in ZK). Is this expected?
>
> Also I was wondering if there was a safe way to cancel a currently running
> task or deleing pending tasks?
>
> Regards,
>
> Ash
>
> --
> **
> ** <https://www.canva.com/>Empowering the world to design
> Also, we're
> hiring. Apply here! <https://about.canva.com/careers/>
>
> <https://twitter.com/canva> <https://facebook.com/canva>
> <https://au.linkedin.com/company/canva> <https://twitter.com/canva>
> <https://facebook.com/canva> <https://au.linkedin.com/company/canva>
> <https://instagram.com/canva>
>
>
>
>
>
>
>
>
>
>