You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by sunnyfr <jo...@gmail.com> on 2009/03/27 01:32:13 UTC
Re: "Unable to move index file" error during replication
Hi,
It doesn't seem to work for me, I changed as well this part below is it ok??
> - List<String> copiedfiles = new ArrayList<String>();
> + Set<String> filesToCopy = new HashSet<String>();
http://www.nabble.com/file/p22734005/ReplicationHandler.java
ReplicationHandler.java
Thanks a lot,
Noble Paul നോബിള് नोब्ळ् wrote:
>
> James thanks .
>
> If this is true the place to fix this is in
> ReplicationHandler#getFileList(). patch is attached.
>
>
> On Wed, Dec 24, 2008 at 4:04 PM, James Grant <Ja...@semantico.com>
> wrote:
>> I had the same problem. It turned out that the list of files from the
>> master
>> included duplicates. When the slave completes the download and tries to
>> move
>> the files into the index it comes across a file that does not exist
>> because
>> it has already been moved so it backs out the whole operation.
>>
>> My solution for now was to patch the copyindexFiles method of
>> org.apache.solr.handler.SnapPuller so that it normalises the list before
>> moving the files. This isn't the best solution since it will still
>> download
>> the file twice but it was the easiest and smallest change to make. The
>> patch
>> is below
>>
>> Regards
>>
>> James
>>
>> --- src/java/org/apache/solr/handler/SnapPuller.java (revision 727347)
>> +++ src/java/org/apache/solr/handler/SnapPuller.java (working copy)
>> @@ -470,7 +470,7 @@
>> */
>> private boolean copyIndexFiles(File snapDir, File indexDir) {
>> String segmentsFile = null;
>> - List<String> copiedfiles = new ArrayList<String>();
>> + Set<String> filesToCopy = new HashSet<String>();
>> for (Map<String, Object> f : filesDownloaded) {
>> String fname = (String) f.get(NAME);
>> // the segments file must be copied last
>> @@ -482,6 +482,10 @@
>> segmentsFile = fname;
>> continue;
>> }
>> + filesToCopy.add(fname);
>> + }
>> + List<String> copiedfiles = new ArrayList<String>();
>> + for (String fname: filesToCopy) {
>> if (!copyAFile(snapDir, indexDir, fname, copiedfiles)) return false;
>> copiedfiles.add(fname);
>> }
>>
>>
>> Jaco wrote:
>>>
>>> Hello,
>>>
>>> While testing out the new replication features, I'm running into some
>>> strange problem. On the slave, I keep getting an error like this after
>>> all
>>> files have been copied from the master to the temporary index.xxxxxxxxx
>>> directory:
>>>
>>> SEVERE: Unable to move index file from:
>>> D:\Data\solr\Slave\data\index.20081224110855\_21e.tvx to:
>>> D:\Data\Solr\Slave\data\index\_21e.tvx
>>>
>>> The replication then stops, index remains in original state, so the
>>> updates
>>> are not available at the slave.
>>>
>>> This is my replication config at the master:
>>>
>>> <requestHandler name="/replication" class="solr.ReplicationHandler" >
>>> <lst name="master">
>>> <!--Replicate on 'optimize' it can also be 'commit' -->
>>> <str name="replicateAfter">commit</str>
>>> <str name="confFiles">schema.xml</str>
>>> </lst>
>>> </requestHandler>
>>>
>>> This is the replication config at the slave:
>>>
>>> <requestHandler name="/replication" class="solr.ReplicationHandler" >
>>> <lst name="slave">
>>> <str name="masterUrl">
>>> http://hostnamemaster:8080/solr/Master/replication</str>
>>> <str name="pollInterval">00:10:00</str>
>>> <str name="zip">true</str>
>>> </lst>
>>> </requestHandler>
>>>
>>> I'm running a Solr nightly build of 21.12.2008 in Tomcat 6 on Windows
>>> 2003.
>>> Initially I thought there was some problem with disk space, but this is
>>> not
>>> the case. Replication did run fine for intial version of index, but
>>> after
>>> that at some point it didn't work anymore. Any ideas what could be wrong
>>> here?
>>>
>>> Thanks very much in advance, bye,
>>>
>>> Jaco.
>>>
>>>
>>
>>
>
>
>
> --
> --Noble Paul
>
> Index: src/java/org/apache/solr/handler/ReplicationHandler.java
> ===================================================================
> --- src/java/org/apache/solr/handler/ReplicationHandler.java (revision
> 729282)
> +++ src/java/org/apache/solr/handler/ReplicationHandler.java (working
> copy)
> @@ -268,7 +268,7 @@
> List<Map<String, Object>> result = new ArrayList<Map<String,
> Object>>();
> try {
> //get all the files in the commit
> - Collection<String> files = commit.getFileNames();
> + Collection<String> files = new
> HashSet<String>(commit.getFileNames());
> for (String fileName : files) {
> File file = new File(core.getIndexDir(), fileName);
> Map<String, Object> fileMeta = getFileInfo(file);
>
>
--
View this message in context: http://www.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-tp21157722p22734005.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: "Unable to move index file" error during replication
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
the latest nightly should do fine
On Fri, Mar 27, 2009 at 1:59 PM, sunnyfr <jo...@gmail.com> wrote:
>
> Sorry but which one shoud I take??
> where exactly ?
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
>>
>> this fix is there in the trunk ,
>> you may not need to apply the patch
>>
>> On Fri, Mar 27, 2009 at 6:02 AM, sunnyfr <jo...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> It doesn't seem to work for me, I changed as well this part below is it
>>> ok??
>>>> - List<String> copiedfiles = new ArrayList<String>();
>>>> + Set<String> filesToCopy = new HashSet<String>();
>>>
>>> http://www.nabble.com/file/p22734005/ReplicationHandler.java
>>> ReplicationHandler.java
>>>
>>> Thanks a lot,
>>>
>>>
>>>
>>>
>>>
>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>>
>>>> James thanks .
>>>>
>>>> If this is true the place to fix this is in
>>>> ReplicationHandler#getFileList(). patch is attached.
>>>>
>>>>
>>>> On Wed, Dec 24, 2008 at 4:04 PM, James Grant <Ja...@semantico.com>
>>>> wrote:
>>>>> I had the same problem. It turned out that the list of files from the
>>>>> master
>>>>> included duplicates. When the slave completes the download and tries to
>>>>> move
>>>>> the files into the index it comes across a file that does not exist
>>>>> because
>>>>> it has already been moved so it backs out the whole operation.
>>>>>
>>>>> My solution for now was to patch the copyindexFiles method of
>>>>> org.apache.solr.handler.SnapPuller so that it normalises the list
>>>>> before
>>>>> moving the files. This isn't the best solution since it will still
>>>>> download
>>>>> the file twice but it was the easiest and smallest change to make. The
>>>>> patch
>>>>> is below
>>>>>
>>>>> Regards
>>>>>
>>>>> James
>>>>>
>>>>> --- src/java/org/apache/solr/handler/SnapPuller.java (revision
>>>>> 727347)
>>>>> +++ src/java/org/apache/solr/handler/SnapPuller.java (working copy)
>>>>> @@ -470,7 +470,7 @@
>>>>> */
>>>>> private boolean copyIndexFiles(File snapDir, File indexDir) {
>>>>> String segmentsFile = null;
>>>>> - List<String> copiedfiles = new ArrayList<String>();
>>>>> + Set<String> filesToCopy = new HashSet<String>();
>>>>> for (Map<String, Object> f : filesDownloaded) {
>>>>> String fname = (String) f.get(NAME);
>>>>> // the segments file must be copied last
>>>>> @@ -482,6 +482,10 @@
>>>>> segmentsFile = fname;
>>>>> continue;
>>>>> }
>>>>> + filesToCopy.add(fname);
>>>>> + }
>>>>> + List<String> copiedfiles = new ArrayList<String>();
>>>>> + for (String fname: filesToCopy) {
>>>>> if (!copyAFile(snapDir, indexDir, fname, copiedfiles)) return
>>>>> false;
>>>>> copiedfiles.add(fname);
>>>>> }
>>>>>
>>>>>
>>>>> Jaco wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> While testing out the new replication features, I'm running into some
>>>>>> strange problem. On the slave, I keep getting an error like this after
>>>>>> all
>>>>>> files have been copied from the master to the temporary
>>>>>> index.xxxxxxxxx
>>>>>> directory:
>>>>>>
>>>>>> SEVERE: Unable to move index file from:
>>>>>> D:\Data\solr\Slave\data\index.20081224110855\_21e.tvx to:
>>>>>> D:\Data\Solr\Slave\data\index\_21e.tvx
>>>>>>
>>>>>> The replication then stops, index remains in original state, so the
>>>>>> updates
>>>>>> are not available at the slave.
>>>>>>
>>>>>> This is my replication config at the master:
>>>>>>
>>>>>> <requestHandler name="/replication" class="solr.ReplicationHandler"
>>>>>> >
>>>>>> <lst name="master">
>>>>>> <!--Replicate on 'optimize' it can also be 'commit' -->
>>>>>> <str name="replicateAfter">commit</str>
>>>>>> <str name="confFiles">schema.xml</str>
>>>>>> </lst>
>>>>>> </requestHandler>
>>>>>>
>>>>>> This is the replication config at the slave:
>>>>>>
>>>>>> <requestHandler name="/replication" class="solr.ReplicationHandler"
>>>>>> >
>>>>>> <lst name="slave">
>>>>>> <str name="masterUrl">
>>>>>> http://hostnamemaster:8080/solr/Master/replication</str>
>>>>>> <str name="pollInterval">00:10:00</str>
>>>>>> <str name="zip">true</str>
>>>>>> </lst>
>>>>>> </requestHandler>
>>>>>>
>>>>>> I'm running a Solr nightly build of 21.12.2008 in Tomcat 6 on Windows
>>>>>> 2003.
>>>>>> Initially I thought there was some problem with disk space, but this
>>>>>> is
>>>>>> not
>>>>>> the case. Replication did run fine for intial version of index, but
>>>>>> after
>>>>>> that at some point it didn't work anymore. Any ideas what could be
>>>>>> wrong
>>>>>> here?
>>>>>>
>>>>>> Thanks very much in advance, bye,
>>>>>>
>>>>>> Jaco.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>>
>>>> Index: src/java/org/apache/solr/handler/ReplicationHandler.java
>>>> ===================================================================
>>>> --- src/java/org/apache/solr/handler/ReplicationHandler.java (revision
>>>> 729282)
>>>> +++ src/java/org/apache/solr/handler/ReplicationHandler.java (working
>>>> copy)
>>>> @@ -268,7 +268,7 @@
>>>> List<Map<String, Object>> result = new ArrayList<Map<String,
>>>> Object>>();
>>>> try {
>>>> //get all the files in the commit
>>>> - Collection<String> files = commit.getFileNames();
>>>> + Collection<String> files = new
>>>> HashSet<String>(commit.getFileNames());
>>>> for (String fileName : files) {
>>>> File file = new File(core.getIndexDir(), fileName);
>>>> Map<String, Object> fileMeta = getFileInfo(file);
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-tp21157722p22734005.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>>
>
> --
> View this message in context: http://www.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-tp21157722p22737672.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: "Unable to move index file" error during replication
Posted by sunnyfr <jo...@gmail.com>.
Sorry but which one shoud I take??
where exactly ?
Noble Paul നോബിള് नोब्ळ् wrote:
>
> this fix is there in the trunk ,
> you may not need to apply the patch
>
> On Fri, Mar 27, 2009 at 6:02 AM, sunnyfr <jo...@gmail.com> wrote:
>>
>> Hi,
>>
>> It doesn't seem to work for me, I changed as well this part below is it
>> ok??
>>> - List<String> copiedfiles = new ArrayList<String>();
>>> + Set<String> filesToCopy = new HashSet<String>();
>>
>> http://www.nabble.com/file/p22734005/ReplicationHandler.java
>> ReplicationHandler.java
>>
>> Thanks a lot,
>>
>>
>>
>>
>>
>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>
>>> James thanks .
>>>
>>> If this is true the place to fix this is in
>>> ReplicationHandler#getFileList(). patch is attached.
>>>
>>>
>>> On Wed, Dec 24, 2008 at 4:04 PM, James Grant <Ja...@semantico.com>
>>> wrote:
>>>> I had the same problem. It turned out that the list of files from the
>>>> master
>>>> included duplicates. When the slave completes the download and tries to
>>>> move
>>>> the files into the index it comes across a file that does not exist
>>>> because
>>>> it has already been moved so it backs out the whole operation.
>>>>
>>>> My solution for now was to patch the copyindexFiles method of
>>>> org.apache.solr.handler.SnapPuller so that it normalises the list
>>>> before
>>>> moving the files. This isn't the best solution since it will still
>>>> download
>>>> the file twice but it was the easiest and smallest change to make. The
>>>> patch
>>>> is below
>>>>
>>>> Regards
>>>>
>>>> James
>>>>
>>>> --- src/java/org/apache/solr/handler/SnapPuller.java (revision
>>>> 727347)
>>>> +++ src/java/org/apache/solr/handler/SnapPuller.java (working copy)
>>>> @@ -470,7 +470,7 @@
>>>> */
>>>> private boolean copyIndexFiles(File snapDir, File indexDir) {
>>>> String segmentsFile = null;
>>>> - List<String> copiedfiles = new ArrayList<String>();
>>>> + Set<String> filesToCopy = new HashSet<String>();
>>>> for (Map<String, Object> f : filesDownloaded) {
>>>> String fname = (String) f.get(NAME);
>>>> // the segments file must be copied last
>>>> @@ -482,6 +482,10 @@
>>>> segmentsFile = fname;
>>>> continue;
>>>> }
>>>> + filesToCopy.add(fname);
>>>> + }
>>>> + List<String> copiedfiles = new ArrayList<String>();
>>>> + for (String fname: filesToCopy) {
>>>> if (!copyAFile(snapDir, indexDir, fname, copiedfiles)) return
>>>> false;
>>>> copiedfiles.add(fname);
>>>> }
>>>>
>>>>
>>>> Jaco wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> While testing out the new replication features, I'm running into some
>>>>> strange problem. On the slave, I keep getting an error like this after
>>>>> all
>>>>> files have been copied from the master to the temporary
>>>>> index.xxxxxxxxx
>>>>> directory:
>>>>>
>>>>> SEVERE: Unable to move index file from:
>>>>> D:\Data\solr\Slave\data\index.20081224110855\_21e.tvx to:
>>>>> D:\Data\Solr\Slave\data\index\_21e.tvx
>>>>>
>>>>> The replication then stops, index remains in original state, so the
>>>>> updates
>>>>> are not available at the slave.
>>>>>
>>>>> This is my replication config at the master:
>>>>>
>>>>> <requestHandler name="/replication" class="solr.ReplicationHandler"
>>>>> >
>>>>> <lst name="master">
>>>>> <!--Replicate on 'optimize' it can also be 'commit' -->
>>>>> <str name="replicateAfter">commit</str>
>>>>> <str name="confFiles">schema.xml</str>
>>>>> </lst>
>>>>> </requestHandler>
>>>>>
>>>>> This is the replication config at the slave:
>>>>>
>>>>> <requestHandler name="/replication" class="solr.ReplicationHandler"
>>>>> >
>>>>> <lst name="slave">
>>>>> <str name="masterUrl">
>>>>> http://hostnamemaster:8080/solr/Master/replication</str>
>>>>> <str name="pollInterval">00:10:00</str>
>>>>> <str name="zip">true</str>
>>>>> </lst>
>>>>> </requestHandler>
>>>>>
>>>>> I'm running a Solr nightly build of 21.12.2008 in Tomcat 6 on Windows
>>>>> 2003.
>>>>> Initially I thought there was some problem with disk space, but this
>>>>> is
>>>>> not
>>>>> the case. Replication did run fine for intial version of index, but
>>>>> after
>>>>> that at some point it didn't work anymore. Any ideas what could be
>>>>> wrong
>>>>> here?
>>>>>
>>>>> Thanks very much in advance, bye,
>>>>>
>>>>> Jaco.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>> Index: src/java/org/apache/solr/handler/ReplicationHandler.java
>>> ===================================================================
>>> --- src/java/org/apache/solr/handler/ReplicationHandler.java (revision
>>> 729282)
>>> +++ src/java/org/apache/solr/handler/ReplicationHandler.java (working
>>> copy)
>>> @@ -268,7 +268,7 @@
>>> List<Map<String, Object>> result = new ArrayList<Map<String,
>>> Object>>();
>>> try {
>>> //get all the files in the commit
>>> - Collection<String> files = commit.getFileNames();
>>> + Collection<String> files = new
>>> HashSet<String>(commit.getFileNames());
>>> for (String fileName : files) {
>>> File file = new File(core.getIndexDir(), fileName);
>>> Map<String, Object> fileMeta = getFileInfo(file);
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-tp21157722p22734005.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context: http://www.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-tp21157722p22737672.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: "Unable to move index file" error during replication
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
this fix is there in the trunk ,
you may not need to apply the patch
On Fri, Mar 27, 2009 at 6:02 AM, sunnyfr <jo...@gmail.com> wrote:
>
> Hi,
>
> It doesn't seem to work for me, I changed as well this part below is it ok??
>> - List<String> copiedfiles = new ArrayList<String>();
>> + Set<String> filesToCopy = new HashSet<String>();
>
> http://www.nabble.com/file/p22734005/ReplicationHandler.java
> ReplicationHandler.java
>
> Thanks a lot,
>
>
>
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
>>
>> James thanks .
>>
>> If this is true the place to fix this is in
>> ReplicationHandler#getFileList(). patch is attached.
>>
>>
>> On Wed, Dec 24, 2008 at 4:04 PM, James Grant <Ja...@semantico.com>
>> wrote:
>>> I had the same problem. It turned out that the list of files from the
>>> master
>>> included duplicates. When the slave completes the download and tries to
>>> move
>>> the files into the index it comes across a file that does not exist
>>> because
>>> it has already been moved so it backs out the whole operation.
>>>
>>> My solution for now was to patch the copyindexFiles method of
>>> org.apache.solr.handler.SnapPuller so that it normalises the list before
>>> moving the files. This isn't the best solution since it will still
>>> download
>>> the file twice but it was the easiest and smallest change to make. The
>>> patch
>>> is below
>>>
>>> Regards
>>>
>>> James
>>>
>>> --- src/java/org/apache/solr/handler/SnapPuller.java (revision 727347)
>>> +++ src/java/org/apache/solr/handler/SnapPuller.java (working copy)
>>> @@ -470,7 +470,7 @@
>>> */
>>> private boolean copyIndexFiles(File snapDir, File indexDir) {
>>> String segmentsFile = null;
>>> - List<String> copiedfiles = new ArrayList<String>();
>>> + Set<String> filesToCopy = new HashSet<String>();
>>> for (Map<String, Object> f : filesDownloaded) {
>>> String fname = (String) f.get(NAME);
>>> // the segments file must be copied last
>>> @@ -482,6 +482,10 @@
>>> segmentsFile = fname;
>>> continue;
>>> }
>>> + filesToCopy.add(fname);
>>> + }
>>> + List<String> copiedfiles = new ArrayList<String>();
>>> + for (String fname: filesToCopy) {
>>> if (!copyAFile(snapDir, indexDir, fname, copiedfiles)) return false;
>>> copiedfiles.add(fname);
>>> }
>>>
>>>
>>> Jaco wrote:
>>>>
>>>> Hello,
>>>>
>>>> While testing out the new replication features, I'm running into some
>>>> strange problem. On the slave, I keep getting an error like this after
>>>> all
>>>> files have been copied from the master to the temporary index.xxxxxxxxx
>>>> directory:
>>>>
>>>> SEVERE: Unable to move index file from:
>>>> D:\Data\solr\Slave\data\index.20081224110855\_21e.tvx to:
>>>> D:\Data\Solr\Slave\data\index\_21e.tvx
>>>>
>>>> The replication then stops, index remains in original state, so the
>>>> updates
>>>> are not available at the slave.
>>>>
>>>> This is my replication config at the master:
>>>>
>>>> <requestHandler name="/replication" class="solr.ReplicationHandler" >
>>>> <lst name="master">
>>>> <!--Replicate on 'optimize' it can also be 'commit' -->
>>>> <str name="replicateAfter">commit</str>
>>>> <str name="confFiles">schema.xml</str>
>>>> </lst>
>>>> </requestHandler>
>>>>
>>>> This is the replication config at the slave:
>>>>
>>>> <requestHandler name="/replication" class="solr.ReplicationHandler" >
>>>> <lst name="slave">
>>>> <str name="masterUrl">
>>>> http://hostnamemaster:8080/solr/Master/replication</str>
>>>> <str name="pollInterval">00:10:00</str>
>>>> <str name="zip">true</str>
>>>> </lst>
>>>> </requestHandler>
>>>>
>>>> I'm running a Solr nightly build of 21.12.2008 in Tomcat 6 on Windows
>>>> 2003.
>>>> Initially I thought there was some problem with disk space, but this is
>>>> not
>>>> the case. Replication did run fine for intial version of index, but
>>>> after
>>>> that at some point it didn't work anymore. Any ideas what could be wrong
>>>> here?
>>>>
>>>> Thanks very much in advance, bye,
>>>>
>>>> Jaco.
>>>>
>>>>
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>> Index: src/java/org/apache/solr/handler/ReplicationHandler.java
>> ===================================================================
>> --- src/java/org/apache/solr/handler/ReplicationHandler.java (revision
>> 729282)
>> +++ src/java/org/apache/solr/handler/ReplicationHandler.java (working
>> copy)
>> @@ -268,7 +268,7 @@
>> List<Map<String, Object>> result = new ArrayList<Map<String,
>> Object>>();
>> try {
>> //get all the files in the commit
>> - Collection<String> files = commit.getFileNames();
>> + Collection<String> files = new
>> HashSet<String>(commit.getFileNames());
>> for (String fileName : files) {
>> File file = new File(core.getIndexDir(), fileName);
>> Map<String, Object> fileMeta = getFileInfo(file);
>>
>>
>
> --
> View this message in context: http://www.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-tp21157722p22734005.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul