You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by ranjitr <co...@yahoo.com> on 2010/02/04 04:05:06 UTC
Solr 1.4: Full import FileNotFoundException
Hello,
I have noticed that when I run concurrent full-imports using DIH in Solr
1.4, the index ends up getting corrupted. I see the following in the log
files (a snippet):
<record>
<date>2010-02-03T17:54:24</date>
<millis>1265248464553</millis>
<sequence>764</sequence>
<logger>org.apache.solr.handler.dataimport.SolrWriter</logger>
<level>SEVERE</level>
<class>org.apache.solr.handler.dataimport.SolrWriter</class>
<method>commit</method>
<thread>25</thread>
<message>Exception while solr commit.</message>
<exception>
<message>java.io.FileNotFoundException:
/solrserver/apache-solr-1.3.0/exampl
e/multicore/RET/data/index/_5.cfs (No such file or directory)</message>
<frame>
<class>java.io.RandomAccessFile</class>
<method>open</method>
</frame>
<frame>
<class>java.io.RandomAccessFile</class>
<method><init></method>
<line>212</line>
</frame>
<frame>
<class>org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor</class>
<method><init></method>
<line>552</line>
</frame>
<frame>
<class>org.apache.lucene.store.FSDirectory$FSIndexInput</class>
<method><init></method>
<line>582</line>
</frame>
<frame>
<class>org.apache.lucene.store.FSDirectory</class>
<method>openInput</method>
<line>488</line>
</frame>
<frame>
<class>org.apache.lucene.index.CompoundFileReader</class>
<method><init></method>
<line>70</line>
</frame>
<frame>
<class>org.apache.lucene.index.SegmentReader</class>
<method>initialize</method>
<line>319</line>
</frame>
<frame>
<class>org.apache.lucene.index.SegmentReader</class>
<method>get</method>
<line>304</line>
</frame>
<frame>
<class>org.apache.lucene.index.SegmentReader</class>
<method>get</method>
<line>234</line>
</frame>
<frame>
<class>org.apache.solr.handler.dataimport.DataImporter$1</class>
<method>run</method>
<line>377</line>
</frame>
</exception>
</record>
Could this be because the concurrent full-imports are stepping on each
other's toes? It seems like one full-import request ends up deleting
another's segment files.
Is there a way to avoid this? Perhaps a config option? I would like to
retain the flexibility to issue concurrent full-import requests.
I found some documentation on this issue at:
http://old.nabble.com/FileNotFoundException-on-index-td25717530.html
But I looked at:
http://old.nabble.com/dataimporthandler-and-multiple-delta-import-td19160129.html
and was under the impression that this issue was fixed in Solr 1.4.
Kindly advise.
Ranjit.
--
View this message in context: http://old.nabble.com/Solr-1.4%3A-Full-import-FileNotFoundException-tp27446982p27446982.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.4: Full import FileNotFoundException
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
can we confirm that the user does not have multiple DIH configured?
any request for an import, while an import is going on, is rejected
On Sat, Feb 13, 2010 at 11:40 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : concurrent imports are not allowed in DIH, unless u setup multiple DIH instances
>
> Right, but that's not the issue -- the question is wether attemping
> to do so might be causing index corruption (either because of a bug or
> because of some possibly really odd config we currently know nothing about)
>
>
> : > : I have noticed that when I run concurrent full-imports using DIH in Solr
> : > : 1.4, the index ends up getting corrupted. I see the following in the log
> : >
> : > I'm fairly confident that concurrent imports won't work -- but it
> : > shouldn't corrupt your index -- even if the DIH didn't actively check for
> : > this type of situation, the underlying Lucene LockFactory should ensure
> : > that one of the inports "wins" ... you'll need to tell us what kind of
> : > Filesystem you are using, and show us the relevent settings from your
> : > solrconfig (lock type, merge policy, indexDefaults, mainIndex, DIH,
> : > etc...)
> : >
> : > At worst you should get a lock time out exception.
> : >
> : > : But I looked at:
> : > : http://old.nabble.com/dataimporthandler-and-multiple-delta-import-td19160129.html
> : > :
> : > : and was under the impression that this issue was fixed in Solr 1.4.
> : >
> : > ...right, attempting to run two concurrent imports with DIH should cause
> : > the second one to abort immediatley.
> : >
> : >
> : >
> : >
> : > -Hoss
> : >
> : >
> :
> :
> :
> : --
> : -----------------------------------------------------
> : Noble Paul | Systems Architect| AOL | http://aol.com
> :
>
>
>
> -Hoss
>
>
--
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com
Re: Solr 1.4: Full import FileNotFoundException
Posted by Chris Hostetter <ho...@fucit.org>.
: concurrent imports are not allowed in DIH, unless u setup multiple DIH instances
Right, but that's not the issue -- the question is wether attemping
to do so might be causing index corruption (either because of a bug or
because of some possibly really odd config we currently know nothing about)
: > : I have noticed that when I run concurrent full-imports using DIH in Solr
: > : 1.4, the index ends up getting corrupted. I see the following in the log
: >
: > I'm fairly confident that concurrent imports won't work -- but it
: > shouldn't corrupt your index -- even if the DIH didn't actively check for
: > this type of situation, the underlying Lucene LockFactory should ensure
: > that one of the inports "wins" ... you'll need to tell us what kind of
: > Filesystem you are using, and show us the relevent settings from your
: > solrconfig (lock type, merge policy, indexDefaults, mainIndex, DIH,
: > etc...)
: >
: > At worst you should get a lock time out exception.
: >
: > : But I looked at:
: > : http://old.nabble.com/dataimporthandler-and-multiple-delta-import-td19160129.html
: > :
: > : and was under the impression that this issue was fixed in Solr 1.4.
: >
: > ...right, attempting to run two concurrent imports with DIH should cause
: > the second one to abort immediatley.
: >
: >
: >
: >
: > -Hoss
: >
: >
:
:
:
: --
: -----------------------------------------------------
: Noble Paul | Systems Architect| AOL | http://aol.com
:
-Hoss
Re: Solr 1.4: Full import FileNotFoundException
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
concurrent imports are not allowed in DIH, unless u setup multiple DIH instances
On Sat, Feb 13, 2010 at 7:05 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : I have noticed that when I run concurrent full-imports using DIH in Solr
> : 1.4, the index ends up getting corrupted. I see the following in the log
>
> I'm fairly confident that concurrent imports won't work -- but it
> shouldn't corrupt your index -- even if the DIH didn't actively check for
> this type of situation, the underlying Lucene LockFactory should ensure
> that one of the inports "wins" ... you'll need to tell us what kind of
> Filesystem you are using, and show us the relevent settings from your
> solrconfig (lock type, merge policy, indexDefaults, mainIndex, DIH,
> etc...)
>
> At worst you should get a lock time out exception.
>
> : But I looked at:
> : http://old.nabble.com/dataimporthandler-and-multiple-delta-import-td19160129.html
> :
> : and was under the impression that this issue was fixed in Solr 1.4.
>
> ...right, attempting to run two concurrent imports with DIH should cause
> the second one to abort immediatley.
>
>
>
>
> -Hoss
>
>
--
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com
Re: Solr 1.4: Full import FileNotFoundException
Posted by Chris Hostetter <ho...@fucit.org>.
: I have noticed that when I run concurrent full-imports using DIH in Solr
: 1.4, the index ends up getting corrupted. I see the following in the log
I'm fairly confident that concurrent imports won't work -- but it
shouldn't corrupt your index -- even if the DIH didn't actively check for
this type of situation, the underlying Lucene LockFactory should ensure
that one of the inports "wins" ... you'll need to tell us what kind of
Filesystem you are using, and show us the relevent settings from your
solrconfig (lock type, merge policy, indexDefaults, mainIndex, DIH,
etc...)
At worst you should get a lock time out exception.
: But I looked at:
: http://old.nabble.com/dataimporthandler-and-multiple-delta-import-td19160129.html
:
: and was under the impression that this issue was fixed in Solr 1.4.
...right, attempting to run two concurrent imports with DIH should cause
the second one to abort immediatley.
-Hoss
Re: Solr 1.4: Full import FileNotFoundException
Posted by ranjitr <co...@yahoo.com>.
Hello,
I was just wondering if any one had any suggestions?
Thank you,
Ranjit.
ranjitr wrote:
>
> Hello,
>
> I have noticed that when I run concurrent full-imports using DIH in Solr
> 1.4, the index ends up getting corrupted. I see the following in the log
> files (a snippet):
>
>
> <record>
> <date>2010-02-03T17:54:24</date>
> <millis>1265248464553</millis>
> <sequence>764</sequence>
> <logger>org.apache.solr.handler.dataimport.SolrWriter</logger>
> <level>SEVERE</level>
> <class>org.apache.solr.handler.dataimport.SolrWriter</class>
> <method>commit</method>
> <thread>25</thread>
> <message>Exception while solr commit.</message>
> <exception>
> <message>java.io.FileNotFoundException:
> /solrserver/apache-solr-1.3.0/exampl
> e/multicore/RET/data/index/_5.cfs (No such file or directory)</message>
> <frame>
> <class>java.io.RandomAccessFile</class>
> <method>open</method>
> </frame>
> <frame>
> <class>java.io.RandomAccessFile</class>
> <method><init></method>
> <line>212</line>
> </frame>
> <frame>
>
> <class>org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor</class>
> <method><init></method>
> <line>552</line>
> </frame>
> <frame>
> <class>org.apache.lucene.store.FSDirectory$FSIndexInput</class>
> <method><init></method>
> <line>582</line>
> </frame>
> <frame>
> <class>org.apache.lucene.store.FSDirectory</class>
> <method>openInput</method>
> <line>488</line>
> </frame>
> <frame>
> <class>org.apache.lucene.index.CompoundFileReader</class>
> <method><init></method>
> <line>70</line>
> </frame>
> <frame>
> <class>org.apache.lucene.index.SegmentReader</class>
> <method>initialize</method>
> <line>319</line>
> </frame>
> <frame>
> <class>org.apache.lucene.index.SegmentReader</class>
> <method>get</method>
> <line>304</line>
> </frame>
> <frame>
> <class>org.apache.lucene.index.SegmentReader</class>
> <method>get</method>
> <line>234</line>
> </frame>
> <frame>
> <class>org.apache.solr.handler.dataimport.DataImporter$1</class>
> <method>run</method>
> <line>377</line>
> </frame>
> </exception>
> </record>
>
>
> Could this be because the concurrent full-imports are stepping on each
> other's toes? It seems like one full-import request ends up deleting
> another's segment files.
>
> Is there a way to avoid this? Perhaps a config option? I would like to
> retain the flexibility to issue concurrent full-import requests.
>
> I found some documentation on this issue at:
> http://old.nabble.com/FileNotFoundException-on-index-td25717530.html
>
> But I looked at:
> http://old.nabble.com/dataimporthandler-and-multiple-delta-import-td19160129.html
>
> and was under the impression that this issue was fixed in Solr 1.4.
>
> Kindly advise.
>
> Ranjit.
>
--
View this message in context: http://old.nabble.com/Solr-1.4%3A-Full-import-FileNotFoundException-tp27446982p27472311.html
Sent from the Solr - User mailing list archive at Nabble.com.