You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Robert Engels <re...@ix.netcom.com> on 2007/01/24 17:04:51 UTC
Re: [jira] Commented: (LUCENE-710) Implement
Curious, I guess I don't understand the BSD disclaimer. The application should not need to track any of this. The OS should be tracking open FD and locks for the process, and when it closes a FD on behalf of a process it should also remove the locks.
-----Original Message-----
>From: "Marvin Humphrey (JIRA)" <ji...@apache.org>
>Sent: Jan 23, 2007 10:56 PM
>To: java-dev@lucene.apache.org
>Subject: [jira] Commented: (LUCENE-710) Implement "point in time" searching without relying on filesystem semantics
>
>
> [ https://issues.apache.org/jira/browse/LUCENE-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466911 ]
>
>Marvin Humphrey commented on LUCENE-710:
>----------------------------------------
>
>On Jan 23, 2007, at 2:19 PM, Michael McCandless (JIRA) wrote:
>
>> First do no harm.
>
>If that was really your guiding philosophy, you would never change anything.
>
>> And Sun's Javadocs on the equivalent Java method, File.createNewFile, has a
>> warning about not relying on this for locking:
>>
>> http://java.sun.com/j2se/1.4.2/docs/api/java/io/File.html#createNewFile()
>
>That page recommends that you use FileLock instead, which maps to Fcntl on
>some systems. The FreeBSD manpage on Fcntl uses less delicate language than
>Sun in pointing out the drawbacks:
>
> This interface follows the completely stupid semantics of System V and
> IEEE Std 1003.1-1988 (``POSIX.1'') that require that all locks associated
> with a file for a given process are removed when any file descriptor for
> that file is closed by that process. This semantic means that applica-
> tions must be aware of any files that a subroutine library may access.
>
>Trying to guarantee that kind of discipline from library code severely limits
>your options.
>
>> This warning is why we created the NativeFSLockFactory for Directory locking
>> in the first place.
>
>Take a look at this bug, which explains how that warning got added.
>
>http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4676183
>
>Read the comment below -- the problem with the "protocol" they warn you
>against using is with deleteOnExit(), not createNewFile(). I think you're
>better off with dot-locks.
>
>> OK. You could implement this in Lucene as a custom deletion policy once we
>> get this commmitted (I think this is 6 proposals now for "deletion policy"
>> for NFS), plus a wrapper around IndexReader.
>
>This was the response I got on the KinoSearch list:
>
> We do not enable NFS writes, only reads (which is why Slashdot is able to
> reliably use NFS for its heavy load :-). So I don't think that will work,
> if I understand you correctly.
>
>Lack of bulletproof support for NFS ain't gonna hold up my next release any
>longer. What a freakin' nightmare...
>
>> Implement "point in time" searching without relying on filesystem semantics
>> ---------------------------------------------------------------------------
>>
>> Key: LUCENE-710
>> URL: https://issues.apache.org/jira/browse/LUCENE-710
>> Project: Lucene - Java
>> Issue Type: Improvement
>> Components: Index
>> Affects Versions: 2.1
>> Reporter: Michael McCandless
>> Assigned To: Michael McCandless
>> Priority: Minor
>>
>> This was touched on in recent discussion on dev list:
>> http://www.gossamer-threads.com/lists/lucene/java-dev/41700#41700
>> and then more recently on the user list:
>> http://www.gossamer-threads.com/lists/lucene/java-user/42088
>> Lucene's "point in time" searching currently relies on how the
>> underlying storage handles deletion files that are held open for
>> reading.
>> This is highly variable across filesystems. For example, UNIX-like
>> filesystems usually do "close on last delete", and Windows filesystem
>> typically refuses to delete a file open for reading (so Lucene retries
>> later). But NFS just removes the file out from under the reader, and
>> for that reason "point in time" searching doesn't work on NFS
>> (see LUCENE-673 ).
>> With the lockless commits changes (LUCENE-701 ), it's quite simple to
>> re-implement "point in time searching" so as to not rely on filesystem
>> semantics: we can just keep more than the last segments_N file (as
>> well as all files they reference).
>> This is also in keeping with the design goal of "rely on as little as
>> possible from the filesystem". EG with lockless we no longer re-use
>> filenames (don't rely on filesystem cache being coherent) and we no
>> longer use file renaming (because on Windows it can fails). This
>> would be another step of not relying on semantics of "deleting open
>> files". The less we require from filesystem the more portable Lucene
>> will be!
>> Where it gets interesting is what "policy" we would then use for
>> removing segments_N files. The policy now is "remove all but the last
>> one". I think we would keep this policy as the default. Then you
>> could imagine other policies:
>> * Keep past N day's worth
>> * Keep the last N
>> * Keep only those in active use by a reader somewhere (note: tricky
>> how to reliably figure this out when readers have crashed, etc.)
>> * Keep those "marked" as rollback points by some transaction, or
>> marked explicitly as a "snaphshot".
>> * Or, roll your own: the "policy" would be an interface or abstract
>> class and you could make your own implementation.
>> I think for this issue we could just create the framework
>> (interface/abstract class for "policy" and invoke it from
>> IndexFileDeleter) and then implement the current policy (delete all
>> but most recent segments_N) as the default policy.
>> In separate issue(s) we could then create the above more interesting
>> policies.
>> I think there are some important advantages to doing this:
>> * "Point in time" searching would work on NFS (it doesn't now
>> because NFS doesn't do "delete on last close"; see LUCENE-673 )
>> and any other Directory implementations that don't work
>> currently.
>> * Transactional semantics become a possibility: you can set a
>> snapshot, do a bunch of stuff to your index, and then rollback to
>> the snapshot at a later time.
>> * If a reader crashes or machine gets rebooted, etc, it could choose
>> to re-open the snapshot it had previously been using, whereas now
>> the reader must always switch to the last commit point.
>> * Searchers could search the same snapshot for follow-on actions.
>> Meaning, user does search, then next page, drill down (Solr),
>> drill up, etc. These are each separate trips to the server and if
>> searcher has been re-opened, user can get inconsistent results (=
>> lost trust). But with, one series of search interactions could
>> explicitly stay on the snapshot it had started with.
>
>--
>This message is automatically generated by JIRA.
>-
>You can reply to this email to add a comment to the issue online.
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-710) Implement
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Jan 24, 2007, at 8:04 AM, Robert Engels wrote:
> Curious, I guess I don't understand the BSD disclaimer. The
> application should not need to track any of this. The OS should be
> tracking open FD and locks for the process, and when it closes a FD
> on behalf of a process it should also remove the locks.
You're right. A less aggressively truncated excerpt should make
things clear. Here's the whole paragraph:
This interface follows the completely stupid semantics of
System V and
IEEE Std 1003.1-1988 (``POSIX.1'') that require that all locks
associated
with a file for a given process are removed when any file
descriptor for
that file is closed by that process. This semantic means that
applica-
tions must be aware of any files that a subroutine library may
access.
For example if an application for updating the password file
locks the
password file database while making the update, and then calls
getpwname(3) to retrieve a record, the lock will be lost because
getpwname(3) opens, reads, and closes the password database.
The data-
base close will release all locks that the process has
associated with
the database, even if the library routine never requested a
lock on the
database. Another minor semantic problem with this interface
is that
locks are not inherited by a child process created using the
fork(2)
function. The flock(2) interface has much more rational last
close
semantics and allows locks to be inherited by child processes.
Flock(2)
is recommended for applications that want to ensure the
integrity of
their locks when using library routines or wish to pass locks
to their
children. Note that flock(2) and fcntl(2) locks may be safely
used con-
currently.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org