Collection backup fails with java.nio.file.NoSuchFileException

[Ere Maijala <er...@helsinki.fi>]

I'm running a three-node SolrCloud (tested with versions 6.4.2 and 
6.5.0) with 1 shard and 3 replicas, and I'm having trouble getting the 
collection backup API to actually do the job. This is the request I use 
to initiate the backup:

http://localhost:8983/solr/admin/collections?action=BACKUP&name=biblioprod&collection=biblio1&location=/data/backup/solr

The result is always something like this:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">500</int><int 
name="QTime">840</int></lst><lst name="failure"><str 
name="[servername]:8983_solr">org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error 
from server at http://[servername]:8983/solr: Failed to backup 
core=biblio1_shard1_replica3 because java.nio.file.NoSuchFileException: 
/data/solr1-1/vufind/biblio1_shard1_replica3/data/index.20170424115540663/segments_y1</str></lst><str 
name="Operation backup caused 
exception:">org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: 
Could not backup all replicas</str><lst name="exception"><str 
name="msg">Could not backup all replicas</str><int 
name="rspCode">500</int></lst><lst name="error"><lst 
name="metadata"><str 
name="error-class">org.apache.solr.common.SolrException</str><str 
name="root-error-class">org.apache.solr.common.SolrException</str></lst><str 
name="msg">Could not backup all replicas</str><str 
name="trace">org.apache.solr.common.SolrException: Could not backup all 
replicas
	at 
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:287)
[snip stack]
</str><int name="code">500</int></lst>
</response>

The index seems to be working fine. I've also tested optimizing it first 
(shouldn't need to do that, right?), but it always fails with 
segments_[something] missing. Also going to Solr Admin UI, selecting the 
collection and clicking e.g. Schema in left hand menu causes the 
following to be written into solr.log:

2017-04-27 11:19:50.724 WARN  (qtp1205044462-1590) [c:biblio1 s:shard1 
r:core_node1 x:biblio1_shard1_replica3] o.a.s.h.a.LukeRequestHandler 
Error getting file length for [segments_y1]
java.nio.file.NoSuchFileException: 
/data/solr1-1/vufind/biblio1_shard1_replica3/data/index.20170424115540663/segments_y1
         at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
         at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
         at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
         at 
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
         at 
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
         at 
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
         at java.nio.file.Files.readAttributes(Files.java:1737)
         at java.nio.file.Files.size(Files.java:2332)
         at 
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
         at 
org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:128)
         at 
org.apache.solr.handler.admin.LukeRequestHandler.getFileLength(LukeRequestHandler.java:614)
         at 
org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:587)
         at 
org.apache.solr.handler.admin.LukeRequestHandler.handleRequestBody(LukeRequestHandler.java:138)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
[snip]

Looks like the backup only works intermittently after the server has 
been sitting for a while without any update requests being made.

Any ideas?

Regards,
Ere