solr backups failing

[r ohara <ro...@gmail.com>]

Hi all,
We are using solrcloud 8.11 with 10 shards. We've been getting errors
trying to back up our index using the collections API. Here's the error we
see in some of the nodes when we run the REQUESTSTATUS api.

Failed to backup core=collection_02_replica_t21 because
java.io.IOException: Input/output error

We ran it successfully once, but since then at least one node, but not
always the same one fails. It is a large index with almost 100 million
documents and 2T total in size. What could cause this error?

Thanks in advance

Re: solr backups failing

[r ohara <ro...@gmail.com>]

The logs did not show anything that would explain the error. I believe
the issue was because I had added a shard since the last backup. Once I
deleted the backup, it worked as expected and subsequent backups worked as
well.

Thanks for the comments.

On Wed, Mar 29, 2023 at 7:10 AM Vivaldi <uy...@vivaldi.net.invalid> wrote:

> Are you using HDFS to backup or shared network directory? In the case of
> shared directory the same location should be writable (accessible and has
> write permissions) from all solr nodes.
>
>
> > On 28 Mar 2023, at 19:22, Houston Putman <ho...@gmail.com>
> wrote:
> >
> > Can you provide more information, such as the request you send to solr
> and
> > logs that might explain the IOException?
> >
> > - Houston
> >
> >> On Wed, Mar 22, 2023 at 10:51 AM r ohara <ro...@gmail.com> wrote:
> >>
> >> Hi all,
> >> We are using solrcloud 8.11 with 10 shards. We've been getting errors
> >> trying to back up our index using the collections API. Here's the error
> we
> >> see in some of the nodes when we run the REQUESTSTATUS api.
> >>
> >> Failed to backup core=collection_02_replica_t21 because
> >> java.io.IOException: Input/output error
> >>
> >> We ran it successfully once, but since then at least one node, but not
> >> always the same one fails. It is a large index with almost 100 million
> >> documents and 2T total in size. What could cause this error?
> >>
> >> Thanks in advance
> >>
>
>

Re: solr backups failing

[Vivaldi <uy...@vivaldi.net.INVALID>]

Are you using HDFS to backup or shared network directory? In the case of shared directory the same location should be writable (accessible and has write permissions) from all solr nodes. 


> On 28 Mar 2023, at 19:22, Houston Putman <ho...@gmail.com> wrote:
> 
> Can you provide more information, such as the request you send to solr and
> logs that might explain the IOException?
> 
> - Houston
> 
>> On Wed, Mar 22, 2023 at 10:51 AM r ohara <ro...@gmail.com> wrote:
>> 
>> Hi all,
>> We are using solrcloud 8.11 with 10 shards. We've been getting errors
>> trying to back up our index using the collections API. Here's the error we
>> see in some of the nodes when we run the REQUESTSTATUS api.
>> 
>> Failed to backup core=collection_02_replica_t21 because
>> java.io.IOException: Input/output error
>> 
>> We ran it successfully once, but since then at least one node, but not
>> always the same one fails. It is a large index with almost 100 million
>> documents and 2T total in size. What could cause this error?
>> 
>> Thanks in advance
>> 

Re: solr backups failing

[Houston Putman <ho...@gmail.com>]

Can you provide more information, such as the request you send to solr and
logs that might explain the IOException?

- Houston

On Wed, Mar 22, 2023 at 10:51 AM r ohara <ro...@gmail.com> wrote:

> Hi all,
> We are using solrcloud 8.11 with 10 shards. We've been getting errors
> trying to back up our index using the collections API. Here's the error we
> see in some of the nodes when we run the REQUESTSTATUS api.
>
> Failed to backup core=collection_02_replica_t21 because
> java.io.IOException: Input/output error
>
> We ran it successfully once, but since then at least one node, but not
> always the same one fails. It is a large index with almost 100 million
> documents and 2T total in size. What could cause this error?
>
> Thanks in advance
>