You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Andrzej Bialecki (JIRA)" <ji...@apache.org> on 2008/11/21 19:47:44 UTC
[jira] Created: (LUCENE-1464) FSDirectory.getDirectory always
creates index path
FSDirectory.getDirectory always creates index path
--------------------------------------------------
Key: LUCENE-1464
URL: https://issues.apache.org/jira/browse/LUCENE-1464
Project: Lucene - Java
Issue Type: Bug
Components: Store
Affects Versions: 2.4, 2.9
Reporter: Andrzej Bialecki
This was reported to me as a Luke bug, but going deeper it proved to be a non-intuitive (broken?) behavior of FSDirectory.
If you use FSDirectory.getDirectory(File nonexistent) on a nonexistent path, but one that is located under some existing parent path, then FSDirectory:174 uses file.mkdirs() to create this directory. One would expect a variant of the method with a boolean flag to decide whether or not to create the output path. However, the API with "create" flag is now deprecated, with a comment that points to IndexWriter's "create" flag. This comment is misleading, because the indicated path is created anyway in the file system just by calling FSDirectory.getDirectory().
I propose to do one of the following:
* reinstate the variant of the method with "create" flag. In case if this flag is false, and the index directory is missing, either return null or throw an IOException,
* keep the API as it is now, but either return null or throw IOException if the index dir is missing. This breaks the backwards compatibility, because now users are required to do file.mkdirs() themselves prior to calling FSDirectory.getDirectory().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1464) FSDirectory.getDirectory always
creates index path
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650199#action_12650199 ]
Michael McCandless commented on LUCENE-1464:
--------------------------------------------
I suppose we could consider adding "boolean readOnly" API to FSDirectory. But I feel like that's overkill. It would also be a problem with the single-instance that the now deprecated FSDir.getDirectory still enforces. I think FSDir should be (remain) a simple conduit.
One improvement we could make is if you try to open an IndexReader on an FSDir for which the directory you specified does not exist, instead of saying "no segments* files found" we could say "directory 'XXX' does not exist"? That's a more transparent error.
The problem is, that'd be a change in the API: instead of returning null from FSDir.list, we'd throw an IOException. Maybe we could do this as part of LUCENE-1468, if we introduce a new "listAll" method.
> FSDirectory.getDirectory always creates index path
> --------------------------------------------------
>
> Key: LUCENE-1464
> URL: https://issues.apache.org/jira/browse/LUCENE-1464
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4, 2.9
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
> Attachments: LUCENE-1464.patch
>
>
> This was reported to me as a Luke bug, but going deeper it proved to be a non-intuitive (broken?) behavior of FSDirectory.
> If you use FSDirectory.getDirectory(File nonexistent) on a nonexistent path, but one that is located under some existing parent path, then FSDirectory:174 uses file.mkdirs() to create this directory. One would expect a variant of the method with a boolean flag to decide whether or not to create the output path. However, the API with "create" flag is now deprecated, with a comment that points to IndexWriter's "create" flag. This comment is misleading, because the indicated path is created anyway in the file system just by calling FSDirectory.getDirectory().
> I propose to do one of the following:
> * reinstate the variant of the method with "create" flag. In case if this flag is false, and the index directory is missing, either return null or throw an IOException,
> * keep the API as it is now, but either return null or throw IOException if the index dir is missing. This breaks the backwards compatibility, because now users are required to do file.mkdirs() themselves prior to calling FSDirectory.getDirectory().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1464) FSDirectory.getDirectory always
creates index path
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649954#action_12649954 ]
Andrzej Bialecki commented on LUCENE-1464:
-------------------------------------------
Ok, after reading LUCENE-773 I reluctantly agree. I think that my doubts are related to the fact that now we can get a valid instance of Directory, which points to an invalid filesystem path, one which perhaps cannot be created at all, and we are going to discover this only when opening IndexWriter with create=true flag. In all other situations we are going to get an IOException about the missing "segments" file, which may be misleading.
For this reason I thought it would be better to test the presence of the index path immediately, and create the directory if necessary in Directory only when it's really needed, i.e. when we intend to use Directory for writing. Essentially, I prefer to distinguish between read-only and read-write Directory on the API level of Directory.
Anyway, as I said, I'm fine with this patch.
> FSDirectory.getDirectory always creates index path
> --------------------------------------------------
>
> Key: LUCENE-1464
> URL: https://issues.apache.org/jira/browse/LUCENE-1464
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4, 2.9
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
> Attachments: LUCENE-1464.patch
>
>
> This was reported to me as a Luke bug, but going deeper it proved to be a non-intuitive (broken?) behavior of FSDirectory.
> If you use FSDirectory.getDirectory(File nonexistent) on a nonexistent path, but one that is located under some existing parent path, then FSDirectory:174 uses file.mkdirs() to create this directory. One would expect a variant of the method with a boolean flag to decide whether or not to create the output path. However, the API with "create" flag is now deprecated, with a comment that points to IndexWriter's "create" flag. This comment is misleading, because the indicated path is created anyway in the file system just by calling FSDirectory.getDirectory().
> I propose to do one of the following:
> * reinstate the variant of the method with "create" flag. In case if this flag is false, and the index directory is missing, either return null or throw an IOException,
> * keep the API as it is now, but either return null or throw IOException if the index dir is missing. This breaks the backwards compatibility, because now users are required to do file.mkdirs() themselves prior to calling FSDirectory.getDirectory().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-1464) FSDirectory.getDirectory always
creates index path
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reassigned LUCENE-1464:
------------------------------------------
Assignee: Michael McCandless
> FSDirectory.getDirectory always creates index path
> --------------------------------------------------
>
> Key: LUCENE-1464
> URL: https://issues.apache.org/jira/browse/LUCENE-1464
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4, 2.9
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
>
> This was reported to me as a Luke bug, but going deeper it proved to be a non-intuitive (broken?) behavior of FSDirectory.
> If you use FSDirectory.getDirectory(File nonexistent) on a nonexistent path, but one that is located under some existing parent path, then FSDirectory:174 uses file.mkdirs() to create this directory. One would expect a variant of the method with a boolean flag to decide whether or not to create the output path. However, the API with "create" flag is now deprecated, with a comment that points to IndexWriter's "create" flag. This comment is misleading, because the indicated path is created anyway in the file system just by calling FSDirectory.getDirectory().
> I propose to do one of the following:
> * reinstate the variant of the method with "create" flag. In case if this flag is false, and the index directory is missing, either return null or throw an IOException,
> * keep the API as it is now, but either return null or throw IOException if the index dir is missing. This breaks the backwards compatibility, because now users are required to do file.mkdirs() themselves prior to calling FSDirectory.getDirectory().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1464) FSDirectory.getDirectory always
creates index path
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649940#action_12649940 ]
Andrzej Bialecki commented on LUCENE-1464:
-------------------------------------------
The patch looks fine to me in the sense that it solves my issue, I'm not sure though if this behavior isn't even less intuitive than before. I'm curious why we shouldn't re-instate the no-create API in Directory?
> FSDirectory.getDirectory always creates index path
> --------------------------------------------------
>
> Key: LUCENE-1464
> URL: https://issues.apache.org/jira/browse/LUCENE-1464
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4, 2.9
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
> Attachments: LUCENE-1464.patch
>
>
> This was reported to me as a Luke bug, but going deeper it proved to be a non-intuitive (broken?) behavior of FSDirectory.
> If you use FSDirectory.getDirectory(File nonexistent) on a nonexistent path, but one that is located under some existing parent path, then FSDirectory:174 uses file.mkdirs() to create this directory. One would expect a variant of the method with a boolean flag to decide whether or not to create the output path. However, the API with "create" flag is now deprecated, with a comment that points to IndexWriter's "create" flag. This comment is misleading, because the indicated path is created anyway in the file system just by calling FSDirectory.getDirectory().
> I propose to do one of the following:
> * reinstate the variant of the method with "create" flag. In case if this flag is false, and the index directory is missing, either return null or throw an IOException,
> * keep the API as it is now, but either return null or throw IOException if the index dir is missing. This breaks the backwards compatibility, because now users are required to do file.mkdirs() themselves prior to calling FSDirectory.getDirectory().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1464) FSDirectory.getDirectory always
creates index path
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1464.
----------------------------------------
Resolution: Fixed
Fix Version/s: 2.9
Committed revision 721670.
I'll take up the "throw more transparent IOException instead of returning null" approach with LUCENE-1468.
> FSDirectory.getDirectory always creates index path
> --------------------------------------------------
>
> Key: LUCENE-1464
> URL: https://issues.apache.org/jira/browse/LUCENE-1464
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4, 2.9
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1464.patch
>
>
> This was reported to me as a Luke bug, but going deeper it proved to be a non-intuitive (broken?) behavior of FSDirectory.
> If you use FSDirectory.getDirectory(File nonexistent) on a nonexistent path, but one that is located under some existing parent path, then FSDirectory:174 uses file.mkdirs() to create this directory. One would expect a variant of the method with a boolean flag to decide whether or not to create the output path. However, the API with "create" flag is now deprecated, with a comment that points to IndexWriter's "create" flag. This comment is misleading, because the indicated path is created anyway in the file system just by calling FSDirectory.getDirectory().
> I propose to do one of the following:
> * reinstate the variant of the method with "create" flag. In case if this flag is false, and the index directory is missing, either return null or throw an IOException,
> * keep the API as it is now, but either return null or throw IOException if the index dir is missing. This breaks the backwards compatibility, because now users are required to do file.mkdirs() themselves prior to calling FSDirectory.getDirectory().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1464) FSDirectory.getDirectory always
creates index path
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649774#action_12649774 ]
Michael McCandless commented on LUCENE-1464:
--------------------------------------------
How about just moving the create logic to the first time createOutput is called?
> FSDirectory.getDirectory always creates index path
> --------------------------------------------------
>
> Key: LUCENE-1464
> URL: https://issues.apache.org/jira/browse/LUCENE-1464
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4, 2.9
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
>
> This was reported to me as a Luke bug, but going deeper it proved to be a non-intuitive (broken?) behavior of FSDirectory.
> If you use FSDirectory.getDirectory(File nonexistent) on a nonexistent path, but one that is located under some existing parent path, then FSDirectory:174 uses file.mkdirs() to create this directory. One would expect a variant of the method with a boolean flag to decide whether or not to create the output path. However, the API with "create" flag is now deprecated, with a comment that points to IndexWriter's "create" flag. This comment is misleading, because the indicated path is created anyway in the file system just by calling FSDirectory.getDirectory().
> I propose to do one of the following:
> * reinstate the variant of the method with "create" flag. In case if this flag is false, and the index directory is missing, either return null or throw an IOException,
> * keep the API as it is now, but either return null or throw IOException if the index dir is missing. This breaks the backwards compatibility, because now users are required to do file.mkdirs() themselves prior to calling FSDirectory.getDirectory().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1464) FSDirectory.getDirectory always
creates index path
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649952#action_12649952 ]
Michael McCandless commented on LUCENE-1464:
--------------------------------------------
I don't understand what you mean by "it's the IndexWriter's job to
create a placeholder for Directory".
It's IndexWriter's job to create a new index. It's FSDir's job to
deal with the particulars of a file system like having to make the
containing directory before creating a file.
With this patch we aren't silently creating dirs. Rather, we create
the dir the first time we need to create a file in that dir. That
seems like the right time to create the dir?
> FSDirectory.getDirectory always creates index path
> --------------------------------------------------
>
> Key: LUCENE-1464
> URL: https://issues.apache.org/jira/browse/LUCENE-1464
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4, 2.9
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
> Attachments: LUCENE-1464.patch
>
>
> This was reported to me as a Luke bug, but going deeper it proved to be a non-intuitive (broken?) behavior of FSDirectory.
> If you use FSDirectory.getDirectory(File nonexistent) on a nonexistent path, but one that is located under some existing parent path, then FSDirectory:174 uses file.mkdirs() to create this directory. One would expect a variant of the method with a boolean flag to decide whether or not to create the output path. However, the API with "create" flag is now deprecated, with a comment that points to IndexWriter's "create" flag. This comment is misleading, because the indicated path is created anyway in the file system just by calling FSDirectory.getDirectory().
> I propose to do one of the following:
> * reinstate the variant of the method with "create" flag. In case if this flag is false, and the index directory is missing, either return null or throw an IOException,
> * keep the API as it is now, but either return null or throw IOException if the index dir is missing. This breaks the backwards compatibility, because now users are required to do file.mkdirs() themselves prior to calling FSDirectory.getDirectory().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1464) FSDirectory.getDirectory always
creates index path
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649950#action_12649950 ]
Andrzej Bialecki commented on LUCENE-1464:
-------------------------------------------
Well, if it's the IndexWriter's job to create a placeholder for Directory, then maybe we shouldn't silently do mkdirs() at all in Directory?
> FSDirectory.getDirectory always creates index path
> --------------------------------------------------
>
> Key: LUCENE-1464
> URL: https://issues.apache.org/jira/browse/LUCENE-1464
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4, 2.9
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
> Attachments: LUCENE-1464.patch
>
>
> This was reported to me as a Luke bug, but going deeper it proved to be a non-intuitive (broken?) behavior of FSDirectory.
> If you use FSDirectory.getDirectory(File nonexistent) on a nonexistent path, but one that is located under some existing parent path, then FSDirectory:174 uses file.mkdirs() to create this directory. One would expect a variant of the method with a boolean flag to decide whether or not to create the output path. However, the API with "create" flag is now deprecated, with a comment that points to IndexWriter's "create" flag. This comment is misleading, because the indicated path is created anyway in the file system just by calling FSDirectory.getDirectory().
> I propose to do one of the following:
> * reinstate the variant of the method with "create" flag. In case if this flag is false, and the index directory is missing, either return null or throw an IOException,
> * keep the API as it is now, but either return null or throw IOException if the index dir is missing. This breaks the backwards compatibility, because now users are required to do file.mkdirs() themselves prior to calling FSDirectory.getDirectory().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1464) FSDirectory.getDirectory always
creates index path
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1464:
---------------------------------------
Attachment: LUCENE-1464.patch
Attached patch that just creates the dir the first time createOutput is called. I plan to commit in a day or so.
> FSDirectory.getDirectory always creates index path
> --------------------------------------------------
>
> Key: LUCENE-1464
> URL: https://issues.apache.org/jira/browse/LUCENE-1464
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4, 2.9
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
> Attachments: LUCENE-1464.patch
>
>
> This was reported to me as a Luke bug, but going deeper it proved to be a non-intuitive (broken?) behavior of FSDirectory.
> If you use FSDirectory.getDirectory(File nonexistent) on a nonexistent path, but one that is located under some existing parent path, then FSDirectory:174 uses file.mkdirs() to create this directory. One would expect a variant of the method with a boolean flag to decide whether or not to create the output path. However, the API with "create" flag is now deprecated, with a comment that points to IndexWriter's "create" flag. This comment is misleading, because the indicated path is created anyway in the file system just by calling FSDirectory.getDirectory().
> I propose to do one of the following:
> * reinstate the variant of the method with "create" flag. In case if this flag is false, and the index directory is missing, either return null or throw an IOException,
> * keep the API as it is now, but either return null or throw IOException if the index dir is missing. This breaks the backwards compatibility, because now users are required to do file.mkdirs() themselves prior to calling FSDirectory.getDirectory().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1464) FSDirectory.getDirectory always
creates index path
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649946#action_12649946 ]
Michael McCandless commented on LUCENE-1464:
--------------------------------------------
bq. I'm curious why we shouldn't re-instate the no-create API in Directory?
See LUCENE-773 for the full explanation.
The short answer is that it's confusing because IndexWriter also has a
create boolean and I think there should be only one. Plus,
IndexWriter's impl of "create=true" is better than FSDir's (it will
retry deleting files if they are still in use, instead of throwing
IOException, and, it will allow the deletion policy to decide what to
do with the older commit points).
> FSDirectory.getDirectory always creates index path
> --------------------------------------------------
>
> Key: LUCENE-1464
> URL: https://issues.apache.org/jira/browse/LUCENE-1464
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.4, 2.9
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
> Attachments: LUCENE-1464.patch
>
>
> This was reported to me as a Luke bug, but going deeper it proved to be a non-intuitive (broken?) behavior of FSDirectory.
> If you use FSDirectory.getDirectory(File nonexistent) on a nonexistent path, but one that is located under some existing parent path, then FSDirectory:174 uses file.mkdirs() to create this directory. One would expect a variant of the method with a boolean flag to decide whether or not to create the output path. However, the API with "create" flag is now deprecated, with a comment that points to IndexWriter's "create" flag. This comment is misleading, because the indicated path is created anyway in the file system just by calling FSDirectory.getDirectory().
> I propose to do one of the following:
> * reinstate the variant of the method with "create" flag. In case if this flag is false, and the index directory is missing, either return null or throw an IOException,
> * keep the API as it is now, but either return null or throw IOException if the index dir is missing. This breaks the backwards compatibility, because now users are required to do file.mkdirs() themselves prior to calling FSDirectory.getDirectory().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org