You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Jason Rutherglen (JIRA)" <ji...@apache.org> on 2009/04/27 22:19:30 UTC
[jira] Created: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Allow setting the IndexWriter docstore to be a different directory
------------------------------------------------------------------
Key: LUCENE-1618
URL: https://issues.apache.org/jira/browse/LUCENE-1618
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Affects Versions: 2.4.1
Reporter: Jason Rutherglen
Priority: Minor
Fix For: 2.9
Add an IndexWriter.setDocStoreDirectory method that allows doc
stores to be placed in a different directory than the IW default
dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703375#action_12703375 ]
Jason Rutherglen commented on LUCENE-1618:
------------------------------------------
{quote}
non-NRT case I can think of is when users make temporary indices in RAM
{quote}
Yes, and there could be others we don't know about.
{quote}
it might be cleaner to make a Directory impl that dispatches certain files to a RAMDir and others to an FSDir
{quote}
Good idea. I'll try that method first. If this one works out, then the API will be public?
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-1618:
-------------------------------------
Attachment: LUCENE-1618.patch
* Copyright added
* CHANGES.txt added
* Cleaned up
* RAMDir specific stuff removed from FSD
{quote}maybe for the "tee" (IndexOutput "writes through" two
Dirs, suggested above) functionality, we should create a
different Directory impl?{quote}
I think a different directory impl makes sense, the
functionality of FileSwitchDirectory is fairly specific.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Eks Dev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703406#action_12703406 ]
Eks Dev commented on LUCENE-1618:
---------------------------------
Maybe,
FileSwitchDirectory should have possibility to get file list/extensions that should be loaded into RAM... making it maintenance free, pushing this decision to end user... if, and when we decide to support users in it, we could than maintain static list at separate place . Kind of separate execution and configuration
I *think* I saw something similar Ning Lee made quite a while ago, from hadoop camp (indexing on hadoop something...). But cannot remember what was it :(
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703676#action_12703676 ]
Yonik Seeley commented on LUCENE-1618:
--------------------------------------
bq. That's not a diff
Sorry, by "diff" I meant the difference in search performance on a RAMDirectory vs NIOFSDirectory where the files are all cached by the OS.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703855#action_12703855 ]
Jason Rutherglen commented on LUCENE-1618:
------------------------------------------
{quote}One downside to this approach is it's brittle - whenever
we change file extensions you'd have to "know" to fix this
Directory.{quote}
True, I don't think we can expect the user to pass in the
correct FileSwitchDirectory (with the attendant file
extensions), we can make the particular implementation of
Directory we use to solve this problem internal to IW. Meaning
the writer can pass through the real directory calls to FSD, and
handle the RAMDir calls on it's own.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707438#action_12707438 ]
Michael McCandless commented on LUCENE-1618:
--------------------------------------------
I think if one is directly writing a file to the primary directory (not through FSD) then one should/could also delete directly from that directory? I don't think we should be putting the magic inside FSD.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703666#action_12703666 ]
Earwin Burrfoot commented on LUCENE-1618:
-----------------------------------------
bq. what is this diff anyway?
That's not a diff, I gave a sample of write-through ram directory Tim and Mike were speaking about.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1618:
---------------------------------------
Attachment: LUCENE-1618.patch
New patch attached w/ minor fixes: added more detail in CHANGES entry; renamed "real" and "other" dir to "primary" and "secondary" dir; tweaked javadocs. I plan to commit later today.
Once this is in, Jason can you update LUCENE-1313 to use this class? Thanks.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1618.
----------------------------------------
Resolution: Fixed
Thanks Jason!
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703684#action_12703684 ]
Earwin Burrfoot commented on LUCENE-1618:
-----------------------------------------
bq. Sorry, by "diff" I meant the difference in search performance on a RAMDirectory vs NIOFSDirectory where the files are all cached by the OS.
Ah! :) It exists. Ranked by speed, directories are FSDirectory (native/sys calls), MMapDirectory (native), RAMDirectory (chunked), MemCachedDirectory (raw array access). But for the purporses of searching a small amount of freshly-indexed docs this difference is miniscule at best, me thinks.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703416#action_12703416 ]
Michael McCandless commented on LUCENE-1618:
--------------------------------------------
{quote}
ileSwitchDirectory should have possibility to get file list/extensions that should be loaded into RAM... making it maintenance free, pushing this decision to end user... if, and when we decide to support users in it, we could than maintain static list at separate place . Kind of separate execution and configuration
{quote}
+1
With flexible indexing, presumably one could use their codec to ask it for the "doc store extensions" vs the "postings extensions", etc., and pass to this configurable FileSwitchDirectory.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703363#action_12703363 ]
Yonik Seeley commented on LUCENE-1618:
--------------------------------------
I can see how this would potentially be useful for realtime... but it seems like only IndexWriter could eventually fix the situation of having the docstore on disk and the rest of a segment in RAM. Which means that this API shouldn't be public?
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703388#action_12703388 ]
Michael McCandless commented on LUCENE-1618:
--------------------------------------------
{quote}
> it might be cleaner to make a Directory impl that dispatches certain files to a RAMDir and others to an FSDir
Good idea. I'll try that method first. If this one works out, then the API will be public?
{quote}
Which API would be public?
If this (call it "FileSwitchDirectory" for now ;) ) works then we would not add any API to IndexWriter (ie it's either or)? But FileSwitchDirectory would be public & "expert".
One downside to this approach is it's brittle -- whenever we change file extensions you'd have to "know" to fix this Directory. Or maybe we make the Directory specialized to only storing the doc stores in the FSDir, then whenever we change file formats we would fix this directory? But in the future, with custom codecs, things could be named whatever... hmmm. Lacking clarity.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reassigned LUCENE-1618:
------------------------------------------
Assignee: Michael McCandless
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Tim Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703630#action_12703630 ]
Tim Smith commented on LUCENE-1618:
-----------------------------------
{quote}
You mean an opened IndexOutput would write its output to two (or more) different places? So you could "write through" a RAMDir down to an FSDir? (This way both the RAMDir and FSDir have a copy of the index).
{quote}
yes, so if you register more than one directory for "index files", then the IndexOutput for the directory would dispatch to an IndexOutput for both sub directories
then, the IndexInput would only be opened on the "primary" directory (for instance, the RAM directory)
This will allow extremely fast searches, with the persistence of a backing FSDirectory
coupled with then having a set of directories for the "Stored Documents", then allows:
* RAM directory search speed
* All changes persisted to disk
* Documents Stored (and retrieved from disk) (or optionally retrieved from RAM)
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703850#action_12703850 ]
Jason Rutherglen commented on LUCENE-1618:
------------------------------------------
{quote}For an NRT writer using RAMDir for recently flushed tiny
segments (LUCENE-1313), the gains are more about the speed of
reading/writing many tiny files. Probably we should try
[somehow] to test this case, to see if LUCENE-1313 is even a
worthwhile optimization.{quote}
True a test would be good, how many files per second would it
produce?
When testing the realtime and the .del files (which are created
numerously before LUCENE-1516) the slowdown was quite dramatic
as it's not a sequential write which means the disk head can
move each time. That coupled with merges going on which
completely ties up the IO I think it's hard for small file
writes to not slow down with a rapidly updating index.
An index that is being updated rapidly presumably would be
performing merges more often to remove deletes.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Earwin Burrfoot updated LUCENE-1618:
------------------------------------
Attachment: MemoryCachedDirectory.java
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703415#action_12703415 ]
Michael McCandless commented on LUCENE-1618:
--------------------------------------------
bq. Would also further suggest that this Directory implementation would take one or more directories to store documents, along with one or more directories to store the index itself
You mean an opened IndexOutput would write its output to two (or more) different places? So you could "write through" a RAMDir down to an FSDir? (This way both the RAMDir and FSDir have a copy of the index).
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703658#action_12703658 ]
Yonik Seeley commented on LUCENE-1618:
--------------------------------------
As it relates to near real time, the search speed of the RAM directory in relation to FSDirectory seems unimportant (what is this diff anyway?) - the FSDirectory will be much larger and that is where the bulk of the search time will be.
It seems like the main benefit of RAMDirectory for NRT is faster creation time (no need to create on-disk files, write them, then sync them), right? Actually the sync is only needed if a new segments file will be written... but there still may be synchronous metadata operations for open-write-close of a file, depending on the FS?
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-1618:
-------------------------------------
Attachment: LUCENE-1618.patch
Added fileExists checking in getDirectory before asking
regarding the extension. This is useful when IndexFileDeleter
uses FSD as a way to combine directories in LUCENE-1313.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703651#action_12703651 ]
Michael McCandless commented on LUCENE-1618:
--------------------------------------------
Neat. This is sounding like one cool Directory...
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704049#action_12704049 ]
Michael McCandless commented on LUCENE-1618:
--------------------------------------------
Patch looks good Jason!
Can you add copyright header & CHANGES.txt entry, and remove some noise (eg TestIndexWriterReader.java)?
Also: I think you should allow any Directory instance as primary/secondary? (You're hardwiring to RAMDir/FSDir now). I realize NRT's use of this will be a RAMDir/FSDir, but I think this dir can be generic. Can you also implement listAll()?
Finally: maybe for the "tee" (IndexOutput "writes through" two Dirs, suggested above) functionality, we should create a different Directory impl?
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703683#action_12703683 ]
Michael McCandless commented on LUCENE-1618:
--------------------------------------------
bq. by "diff" I meant the difference in search performance on a RAMDirectory vs NIOFSDirectory where the files are all cached by the OS.
It's a good question -- I haven't tested it directly. I'd love to know too...
For an NRT writer using RAMDir for recently flushed tiny segments (LUCENE-1313), the gains are more about the speed of reading/writing many tiny files. Probably we should try [somehow] to test this case, to see if LUCENE-1313 is even a worthwhile optimization.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-1618:
-------------------------------------
Attachment: LUCENE-1618.patch
Implementation of the FileSwitchDirectory. It's nice this works
so elegantly with the existing Lucene APIs.
The test case makes sure the fdt and fdx files are written to
the fsdirectory based on the files extensions. I feel that
LUCENE-1313 will depend on this and I'll implement LUCENE-1313
with this patch in mind. I'm not sure how we insure there are no
file name collisions between the real dir and FSD? Because IW is
managing the creation of the segment names I don't think we
need to worry about this.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703370#action_12703370 ]
Michael McCandless commented on LUCENE-1618:
--------------------------------------------
Yeah I also think this should be an "under the hood" (done only by NRT) optimization inside IndexWriter.
The only possible non-NRT case I can think of is when users make temporary indices in RAM, it's possible one would want to write the docStore files to an FSDirectory (because they are so large) but keep postings, norms, deletes, etc in RAM. But going down that road opens up a can of worms... eg does segments_N somehow have to keep track of which dir has which parts of a segment? Suddenly IndexReader must also know to look in different dirs for different parts of a segment, etc.
it might be cleaner to make a Directory impl that dispatches certain files to a RAMDir and others to an FSDir, so IndexWriter/IndexReader still see a single Directory API.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703656#action_12703656 ]
Earwin Burrfoot commented on LUCENE-1618:
-----------------------------------------
bq. You mean an opened IndexOutput would write its output to two (or more) different places?
Except the best way is to write directly to FSDir.IndexOutput, and when it is closed, read back into memory.
That way, if FSDir.IO hits an exception while writing, you don't have to jump through the hoops to keep your RAMDir in consistent state (we had real troubles when some files were 'written' to RAMDir, but failed to persist in FSDir).
Also, when reading the file back you already know it's exact size and can allocate appropriate buffer, saving on resizings (my draft impl) / chunking (lucene's current impl) overhead.
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Tim Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703384#action_12703384 ]
Tim Smith commented on LUCENE-1618:
-----------------------------------
Would also further suggest that this Directory implementation would take one or more directories to store documents, along with one or more directories to store the index itself
one of the directories should be explicitly marked for "reading" for each use
this allows creating a Directory instance that will:
* store documents to disk (reading from disk during searches)
* write index to disk and ram (reading from RAM during searches)
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707284#action_12707284 ]
Michael McCandless commented on LUCENE-1618:
--------------------------------------------
bq. Added fileExists checking in getDirectory
Jason, why is this needed? Why is the mapping based on extension insufficient?
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-1618:
-------------------------------------
Attachment: LUCENE-1618.patch
Added getter methods to FSD for the underlying directories
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707403#action_12707403 ]
Jason Rutherglen commented on LUCENE-1618:
------------------------------------------
One example of the use case is when IndexFileDeleter needs to
access the directory's files as is without extension
interpretation. A .fdt file that was written directly to the
primary directory (not through FSD) would fit this case. When
IFD tries to access the .fdt file (using the current code) FSD
says it's not there (because it thinks it's in the secondary
dir).
Maybe we need a different type of FSD for this case?
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1618) Allow setting the IndexWriter
docstore to be a different directory
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705548#action_12705548 ]
Michael McCandless commented on LUCENE-1618:
--------------------------------------------
OK thanks Jason, I just committed that (w/ small change to listAll to directly allocate the String[]).
> Allow setting the IndexWriter docstore to be a different directory
> ------------------------------------------------------------------
>
> Key: LUCENE-1618
> URL: https://issues.apache.org/jira/browse/LUCENE-1618
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, LUCENE-1618.patch, MemoryCachedDirectory.java
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Add an IndexWriter.setDocStoreDirectory method that allows doc
> stores to be placed in a different directory than the IW default
> dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org