You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Craig Walls <wa...@michaels.com> on 2002/10/11 18:02:16 UTC
FileNotFoundException while indexing
This is my first post to this mailing list, so I hope it works...
We've been trying to use Lucene as our search solution, but every so
often we get a ton of the following in our log files:
java.io.FileNotFoundException:
/usr/WebSphere/michaels/search/working/artprints/_xx.fnm (A file or
directory in the path name does not exist.)
at java.io.RandomAccessFile.open(Native Method)
at
java.io.RandomAccessFile.<init>(RandomAccessFile.java(Compiled Code))
at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
at org.apache.lucene.index.FieldInfos.<init>(Unknown Source)
at org.apache.lucene.index.SegmentReader.<init>(Unknown Source)
at org.apache.lucene.index.IndexWriter.mergeSegments(Unknown
Source)
at
org.apache.lucene.index.IndexWriter.maybeMergeSegments(Unknown Source)
at org.apache.lucene.index.IndexWriter.addDocument(Unknown
Source)
at
com.michaels.search.ProductIndexer.run(ProductIndexer.java:39)
at com.michaels.search.MasterIndexer.run(MasterIndexer.java:56)
at java.lang.Thread.run(Thread.java:512)
At first, we thought it was because we had multiple threads trying to
add to the same index. That concerned me a bit because I thought that
the write.lock and commit.lock files would prevent bad things like this
from happening if multiple threads were writing to the same index.
Nevertheless, we now have a single thread doing our indexing and it
seemed to work fine for several days, but this morning we found the same
errors in the log file.
What would cause this, how can we fix it, and why isn't write.lock and
commit.lock not seeming to help with this?
A bit more info, if it helps: We are running this within a thread that
is kicked off by a servlet when a certain URL is visited. If an indexer
thread is already in progress, we don't kick off another thread. This is
all running within WebSphere 4, running on AIX.
Any help would be greatly appreciated.
Thanks,
Craig
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: FileNotFoundException while indexing
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Regarding final - my guess is that Doug didn't want people to inherit
things that shouldn't be altered, plus I believe at one point in time
Java compiler was relying on 'final' to streamline method calls,
instance vars, classes, etc....to improve performance.
This is a guess.
Otis
--- Stas Chetvertkov <sc...@oilspace.com> wrote:
> Well... there is one thing with my solution. I implemented it using
> native
> code since I found no way to do it in Java :(
>
> Here is an excerpt from 'man 2 creat':
>
> ...O_EXCL is broken on NFS file systems, programs which rely on it
> for
> performing locking tasks will contain a race condition. The solution
> for
> performing atomic file locking using a lockfile is to create a unique
> file
> on the same fs (e.g., incorporating hostname and pid), use link(2)
> to
> make a link to the lockfile. If link() returns 0, the lock is
> successful.
> Otherwise, use stat(2) on the unique file to check if its link count
> has
> increased to 2, in which case the lock is also successful....
>
> This is exactly what I did for locking. I am attaching code that
> solved the
> problem for me (my OS is Linux).
>
> By the way, maybe somebody knows why almost all classes / methods in
> Lucene
> are final? I would surely prefer to inherit from FSDirectory instead
> of
> writing a wrapper around it.
>
> Cheers,
> Stas.
>
> ----- Original Message -----
> From: "Terry Steichen" <te...@net-frame.com>
> To: "Lucene Users List" <lu...@jakarta.apache.org>
> Sent: Friday, October 11, 2002 8:42 PM
> Subject: Re: FileNotFoundException while indexing
>
>
> > Stas,
> >
> > Would you be able/willing to share that improved Directory with the
> list?
> >
> > Regards,
> >
> > Terry
> >
>
>
> ATTACHMENT part 2 application/octet-stream name=FSLockService.java
> ATTACHMENT part 3 application/octet-stream
name=LockingFSDirectory.java
> ATTACHMENT part 4 application/octet-stream
name=FSLockService_NativeLockService.h
> ATTACHMENT part 5 application/octet-stream name=LockService.c
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
__________________________________________________
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: FileNotFoundException while indexing
Posted by Stas Chetvertkov <sc...@oilspace.com>.
Well... there is one thing with my solution. I implemented it using native
code since I found no way to do it in Java :(
Here is an excerpt from 'man 2 creat':
...O_EXCL is broken on NFS file systems, programs which rely on it for
performing locking tasks will contain a race condition. The solution for
performing atomic file locking using a lockfile is to create a unique file
on the same fs (e.g., incorporating hostname and pid), use link(2) to
make a link to the lockfile. If link() returns 0, the lock is successful.
Otherwise, use stat(2) on the unique file to check if its link count has
increased to 2, in which case the lock is also successful....
This is exactly what I did for locking. I am attaching code that solved the
problem for me (my OS is Linux).
By the way, maybe somebody knows why almost all classes / methods in Lucene
are final? I would surely prefer to inherit from FSDirectory instead of
writing a wrapper around it.
Cheers,
Stas.
----- Original Message -----
From: "Terry Steichen" <te...@net-frame.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Friday, October 11, 2002 8:42 PM
Subject: Re: FileNotFoundException while indexing
> Stas,
>
> Would you be able/willing to share that improved Directory with the list?
>
> Regards,
>
> Terry
>
Re: FileNotFoundException while indexing
Posted by Terry Steichen <te...@net-frame.com>.
Stas,
Would you be able/willing to share that improved Directory with the list?
Regards,
Terry
----- Original Message -----
From: "Stas Chetvertkov" <sc...@oilspace.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Friday, October 11, 2002 12:33 PM
Subject: Re: FileNotFoundException while indexing
> I had experienced similar problem with FileNotFoundExceptions.
>
> Our system has one server writing to index (located on NFS mounted disk)
and
> several servers searching in it. In my case, those exceptions were thrown
by
> search servers.
>
> I found out that Lucene locking doesnot work on NFS - in this case 2
> different processes can call createNewFile() for the same file and get
true
> as a result, and Lucene relies on this method. Archiving server was
merging
> segments, this caused deletion of some files, and at the same time search
> server was trying to read those and did not succeed.
>
> I solved this problem by implementing my own Directory with reliable
> makeLock() method.
>
> Cheers,
> Stas.
>
> ----- Original Message -----
> From: "Craig Walls" <wa...@michaels.com>
> To: <lu...@jakarta.apache.org>
> Sent: Friday, October 11, 2002 8:02 PM
> Subject: FileNotFoundException while indexing
>
>
> > This is my first post to this mailing list, so I hope it works...
> >
> > We've been trying to use Lucene as our search solution, but every so
> > often we get a ton of the following in our log files:
> >
> > java.io.FileNotFoundException:
> > /usr/WebSphere/michaels/search/working/artprints/_xx.fnm (A file or
> > directory in the path name does not exist.)
> > at java.io.RandomAccessFile.open(Native Method)
> > at
> > java.io.RandomAccessFile.<init>(RandomAccessFile.java(Compiled Code))
> > at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> > at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> > at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> > at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> > at org.apache.lucene.index.FieldInfos.<init>(Unknown Source)
> > at org.apache.lucene.index.SegmentReader.<init>(Unknown Source)
> > at org.apache.lucene.index.IndexWriter.mergeSegments(Unknown
> > Source)
> > at
> > org.apache.lucene.index.IndexWriter.maybeMergeSegments(Unknown Source)
> > at org.apache.lucene.index.IndexWriter.addDocument(Unknown
> > Source)
> > at
> > com.michaels.search.ProductIndexer.run(ProductIndexer.java:39)
> > at com.michaels.search.MasterIndexer.run(MasterIndexer.java:56)
> > at java.lang.Thread.run(Thread.java:512)
> >
> > At first, we thought it was because we had multiple threads trying to
> > add to the same index. That concerned me a bit because I thought that
> > the write.lock and commit.lock files would prevent bad things like this
> > from happening if multiple threads were writing to the same index.
> > Nevertheless, we now have a single thread doing our indexing and it
> > seemed to work fine for several days, but this morning we found the same
> > errors in the log file.
> >
> > What would cause this, how can we fix it, and why isn't write.lock and
> > commit.lock not seeming to help with this?
> >
> > A bit more info, if it helps: We are running this within a thread that
> > is kicked off by a servlet when a certain URL is visited. If an indexer
> > thread is already in progress, we don't kick off another thread. This is
> > all running within WebSphere 4, running on AIX.
> >
> > Any help would be greatly appreciated.
> > Thanks,
> > Craig
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> <ma...@jakarta.apache.org>
> >
> >
>
>
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: FileNotFoundException while indexing
Posted by Stas Chetvertkov <sc...@oilspace.com>.
I had experienced similar problem with FileNotFoundExceptions.
Our system has one server writing to index (located on NFS mounted disk) and
several servers searching in it. In my case, those exceptions were thrown by
search servers.
I found out that Lucene locking doesnot work on NFS - in this case 2
different processes can call createNewFile() for the same file and get true
as a result, and Lucene relies on this method. Archiving server was merging
segments, this caused deletion of some files, and at the same time search
server was trying to read those and did not succeed.
I solved this problem by implementing my own Directory with reliable
makeLock() method.
Cheers,
Stas.
----- Original Message -----
From: "Craig Walls" <wa...@michaels.com>
To: <lu...@jakarta.apache.org>
Sent: Friday, October 11, 2002 8:02 PM
Subject: FileNotFoundException while indexing
> This is my first post to this mailing list, so I hope it works...
>
> We've been trying to use Lucene as our search solution, but every so
> often we get a ton of the following in our log files:
>
> java.io.FileNotFoundException:
> /usr/WebSphere/michaels/search/working/artprints/_xx.fnm (A file or
> directory in the path name does not exist.)
> at java.io.RandomAccessFile.open(Native Method)
> at
> java.io.RandomAccessFile.<init>(RandomAccessFile.java(Compiled Code))
> at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> at org.apache.lucene.index.FieldInfos.<init>(Unknown Source)
> at org.apache.lucene.index.SegmentReader.<init>(Unknown Source)
> at org.apache.lucene.index.IndexWriter.mergeSegments(Unknown
> Source)
> at
> org.apache.lucene.index.IndexWriter.maybeMergeSegments(Unknown Source)
> at org.apache.lucene.index.IndexWriter.addDocument(Unknown
> Source)
> at
> com.michaels.search.ProductIndexer.run(ProductIndexer.java:39)
> at com.michaels.search.MasterIndexer.run(MasterIndexer.java:56)
> at java.lang.Thread.run(Thread.java:512)
>
> At first, we thought it was because we had multiple threads trying to
> add to the same index. That concerned me a bit because I thought that
> the write.lock and commit.lock files would prevent bad things like this
> from happening if multiple threads were writing to the same index.
> Nevertheless, we now have a single thread doing our indexing and it
> seemed to work fine for several days, but this morning we found the same
> errors in the log file.
>
> What would cause this, how can we fix it, and why isn't write.lock and
> commit.lock not seeming to help with this?
>
> A bit more info, if it helps: We are running this within a thread that
> is kicked off by a servlet when a certain URL is visited. If an indexer
> thread is already in progress, we don't kick off another thread. This is
> all running within WebSphere 4, running on AIX.
>
> Any help would be greatly appreciated.
> Thanks,
> Craig
>
>
>
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>