You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2007/01/12 20:56:27 UTC
[jira] Created: (LUCENE-771) Change default write lock file
location to index directory (not java.io.tmpdir)
Change default write lock file location to index directory (not java.io.tmpdir)
-------------------------------------------------------------------------------
Key: LUCENE-771
URL: https://issues.apache.org/jira/browse/LUCENE-771
Project: Lucene - Java
Issue Type: Improvement
Components: Store
Affects Versions: 2.1
Reporter: Michael McCandless
Assigned To: Michael McCandless
Priority: Minor
Fix For: 2.1
Now that readers are read-only, we no longer need to store lock files
in a different global lock directory than the index directory. This
has been a source of confusion and caused problems to users in the
past.
Furthermore, once the write lock is stored in the index directory, it
no longer needs the big digest prefix that was previously required
to make sure lock files in the global lock directory, from different
indexes, did not conflict.
This way, all files related to an index will appear in a single
directory. And you can easily list that directory to see if a
"write.lock" is present to check whether a writer is open on the
index.
Note that this change just affects how FSDirectory creates its default
lockFactory if no lockFactory was specified. It is still possible
(just no longer the default) to pick a different directory to store
your lock files by pre-instantiating your own LockFactory.
As part of this I would like to remove LOCK_DIR and the no-argument
constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
think we should have the notion of a global default lock directory
anymore. This is actually an API change. However, neither
SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
so I think this API removal is allowed?
Finally I want to deprecate (but not yet remove, because this has been
in the API for many releases) the static LOCK_DIR that's in
FSDirectory. But it's now entirely unused.
See here for discussion leading to this:
http://www.gossamer-threads.com/lists/lucene/java-dev/43940
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-771) Change default write lock file
location to index directory (not java.io.tmpdir)
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464385 ]
Doron Cohen commented on LUCENE-771:
------------------------------------
I have a question on this change - though I didn't look at the code yet - we don't support backwards compatability with regard to existing lock files, right? Specifically, if there exist an older version index with locks in tmp dir, and the Lucene lib is replaced for one client at a time, while other clients are running, there would be a problem, possible corruption. So who ever is migrating from an index before this change to an index with this change should make sure that all clients are upgraded at the time. At least all clients that may attempt to write.
> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-771) Change default write lock file
location to index directory (not java.io.tmpdir)
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-771.
---------------------------------------
Resolution: Fixed
> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Reopened: (LUCENE-771) Change default write lock file
location to index directory (not java.io.tmpdir)
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reopened LUCENE-771:
---------------------------------------
Woops, I committed a silly violation of the LockFactory abstraction
(I'm removing files in FSDirectory when that should only occur inside
the LockFactory implementation.) I'm re-opening and I'll get the
corrected fix in soon.
> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-771) Change default write lock file
location to index directory (not java.io.tmpdir)
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464409 ]
Michael McCandless commented on LUCENE-771:
-------------------------------------------
You're right, backwards compatibility will allow a 2.1 client to
read/write to an older index, but as soon as a 2.1 client writes to
the index then all pre-2.1 readers won't be able to read it. You're
right that if these readers are not reopened then "point in time"
would allow them to keep running pre-2.1.
So I think you could individually update each reader to 2.1, but then
(after readers are updated), all writers would need to go to 2.1 at
once, to make sure they all "agree" that the write lock is now in the
index.
> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-771) Change default write lock file
location to index directory (not java.io.tmpdir)
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464395 ]
Doron Cohen commented on LUCENE-771:
------------------------------------
Is that true? I thought that for previous format changes, the combination of - { (1) point-in-time index reading by readers (2) backwards compatibility (3) locks } - allowed not to require this.
> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-771) Change default write lock file
location to index directory (not java.io.tmpdir)
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-771.
---------------------------------------
Resolution: Fixed
Thanks for the review Yonik!
And thanks for pointing this out Marvin!
I've committed this.
> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-771) Change default write lock file
location to index directory (not java.io.tmpdir)
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464350 ]
Yonik Seeley commented on LUCENE-771:
-------------------------------------
Sounds good, I agree with all the changes you outlined.
> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-771) Change default write lock file
location to index directory (not java.io.tmpdir)
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464393 ]
Michael McCandless commented on LUCENE-771:
-------------------------------------------
Yes, that is true.
But there are also quite a few other changes in 2.1 (eg the file format changes) that would also generally necessitate that all places that will share the given index, upgrade to 2.1 at once.
> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org