Re: Replication snapshot, tar says "file changed as we read it"

[Andrew Clegg <an...@gmail.com>]

Sorry to re-open an old thread, but this just happened to me again,
even with a 30 second sleep between taking the snapshot and starting
to tar it up. Then, even more strangely, the snapshot was removed
again before tar completed.

Archiving snapshot.20110320113401 into
/var/www/mesh/backups/weekly.snapshot.20110320113401.tar.bz2
tar: snapshot.20110320113401/_neqv.fdt: file changed as we read it
tar: snapshot.20110320113401/_neqv.prx: File removed before we read it
tar: snapshot.20110320113401/_neqv.fnm: File removed before we read it
tar: snapshot.20110320113401: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors

Has anybody seen this before, or been able to replicate it themselves?
(no pun intended)

Or, is anyone else using replication snapshots for backup? Have I
misunderstood them? I thought the point of a snapshot was that once
taken it was immutable.

If it's important, this is on a machine configured as a replication
master, but with no slaves attached to it (it's basically a failover
and backup machine).

  <requestHandler name="/replication" class="solr.ReplicationHandler" >
      <lst name="master">
          <str name="replicateAfter">startup</str>
          <str name="replicateAfter">commit</str>
          <str name="confFiles">admin-extra.html,elevate.xml,protwords.txt,schema.xml,scripts.conf,solrconfig_slave.xml:solrconfig.xml,stopwords.txt,synonyms.txt</str>
          <str name="commitReserveDuration">00:00:10</str>
      </lst>
  </requestHandler>

Thanks,

Andrew.


On 16 January 2011 12:55, Andrew Clegg <an...@gmail.com> wrote:
> PS one other point I didn't mention is that this server has a very
> fast autocommit limit (2 seconds max time).
>
> But I don't know if this is relevant -- I thought the files in the
> snapshot wouldn't be committed to again. Please correct me if this is
> a huge misunderstanding.
>
> On 16 January 2011 12:30, Andrew Clegg <an...@gmail.com> wrote:
>> (Many apologies if this appears twice, I tried to send it via Nabble
>> first but it seems to have got stuck, and is fairly urgent/serious.)
>>
>> Hi,
>>
>> I'm trying to use the replication handler to take snapshots, then
>> archive them and ship them off-site.
>>
>> Just now I got a message from tar that worried me:
>>
>> tar: snapshot.20110115035710/_70b.tis: file changed as we read it
>> tar: snapshot.20110115035710: file changed as we read it
>>
>> The relevant bit of script that does it looks like this (error
>> checking removed):
>>
>> curl 'http://localhost:8983/solr/core/1replication?command=backup'
>> PREFIX=''
>> if [[ "$START_TIME" =~ 'Sun' ]]
>> then
>>        PREFIX='weekly.'
>> fi
>> cd $SOLR_DATA_DIR
>> for snapshot in `ls -d -1 snapshot.*`
>> do
>>        TARGET="${LOCAL_BACKUP_DIR}/${PREFIX}${snapshot}.tar.bz2"
>>        echo "Archiving ${snapshot} into $TARGET"
>>        tar jcf $TARGET $snapshot
>>        echo "Deleting ${snapshot}"
>>        rm -rf $snapshot
>> done
>>
>> I was under the impression that files in the snapshot were guaranteed
>> to never change, right? Otherwise what's the point of the replication
>> backup command?
>>
>> I tried putting in a 30-second sleep after the snapshot and before the
>> tar, but the error occurred again anyway.
>>
>> There was a message from Lance N. with a similar error in, years ago:
>>
>> http://www.mail-archive.com/solr-user@lucene.apache.org/msg06104.html
>>
>> but that would be pre-replication anyway, right?
>>
>> This is on Ubuntu 10.10 using java 1.6.0_22 and Solr 1.4.0.
>>
>> Thanks,
>>
>> Andrew.
>>
>>
>> --
>>
>> :: http://biotext.org.uk/ :: http://twitter.com/andrew_clegg/ ::
>>
>
>
>
> --
>
> :: http://biotext.org.uk/ :: http://twitter.com/andrew_clegg/ ::
>



-- 

:: http://biotext.org.uk/ :: http://twitter.com/andrew_clegg/ ::