You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Jason Rutherglen (JIRA)" <ji...@apache.org> on 2009/04/07 03:23:13 UTC
[jira] Created: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
-------------------------------------------------------
Key: LUCENE-1589
URL: https://issues.apache.org/jira/browse/LUCENE-1589
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Affects Versions: 2.4.1
Reporter: Jason Rutherglen
Priority: Minor
Fix For: 2.9
Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
but for IndexReaders. This will be used to flush cloned ram
indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-1589:
-------------------------------------
Attachment: LUCENE-1589.patch
First pass, not committable, no tests yet.
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-1589.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1589:
---------------------------------------
Fix Version/s: (was: 2.9)
3.1
Moving out.
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 3.1
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696812#action_12696812 ]
Jason Rutherglen commented on LUCENE-1589:
------------------------------------------
The deletes are coming into the existing IndexReaders, then we
do the IW.commitMergedDeletes styled copy of new deletes into
the newly merged readers. Are there caveats?
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696650#action_12696650 ]
Jason Rutherglen commented on LUCENE-1589:
------------------------------------------
I started, but because MergePolicy.OneMerge in trunk operates with
SegmentInfos (rather than IndexReaders like LUCENE-1516) this
patch will need wait. It requires being able to pass
readers into OneMerge?
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774668#action_12774668 ]
Michael McCandless commented on LUCENE-1589:
--------------------------------------------
bq. This could probably just throw an exception if the passed in reader is not read-only?
We could do that, but I think it defeats using it for NRT? Though I think it'd still be very useful (and we'd then deprecate the addIndexes(IndexReader[]) that does optimization).
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 3.1
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774360#action_12774360 ]
Michael Busch commented on LUCENE-1589:
---------------------------------------
This could probably just throw an exception if the passed in reader is not read-only?
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 3.1
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696837#action_12696837 ]
Jason Rutherglen commented on LUCENE-1589:
------------------------------------------
I took a walk and thought about this, it seems that we do not
need to write any new code, we can simply ref and clone
addIndexesNoOptimize(IndexReader[] readers) to
MergePolicy.OneMerge.readers/readersClone. Meaning the problem
of new deletes while merging is solved? Of course this would
imply we synchronize on the source reader in commitMergedDeletes
(because deleteDocument is synced?)
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776034#action_12776034 ]
Jason Rutherglen commented on LUCENE-1589:
------------------------------------------
I think we'd need to support read only readers. I'd like to be able to use this API for NRT (and not have the read only reader limitation), however, it would at least require passing a lock object in, and perhaps more.
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 3.1
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader... readers)
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-1589:
----------------------------------
Summary: IndexWriter.addIndexesNoOptimize(IndexReader... readers) (was: IndexWriter.addIndexesNoOptimize(IndexReader[] readers))
Updated for Java 5 and consistency in IndexWriter
> IndexWriter.addIndexesNoOptimize(IndexReader... readers)
> --------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-1589.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12697001#action_12697001 ]
Michael McCandless commented on LUCENE-1589:
--------------------------------------------
{quote}
The deletes are coming into the existing IndexReaders, then we
do the IW.commitMergedDeletes styled copy of new deletes into
the newly merged readers. Are there caveats?
{quote}
I'm now thinking that we should do all of this, internally to IW, under the hood, when it's doing NRT (as part of LUCENE-1313).
Ie, we should not expose an external addIndexes API that must deal with ongoing deletes arriving to the IndexReaders you had passed in.
I think it's useful to expose such an API, with the restriction that you should not be modifying those IR's (deletes, norms) while addIndexes is running. Ie, that method would be just like the addIndexes(IndexReader[]) we have today, but it'd have the same benefits of addIndexesNoOptimize.
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696434#action_12696434 ]
Michael McCandless commented on LUCENE-1589:
--------------------------------------------
Jason are you working on a patch here?
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696643#action_12696643 ]
Jason Rutherglen commented on LUCENE-1589:
------------------------------------------
Yes, because this will block the RAMDir NRT implementation.
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1589)
IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696809#action_12696809 ]
Michael McCandless commented on LUCENE-1589:
--------------------------------------------
Hmm yes. This is also tricky: how would you use this to flush cloned ram indexes? Can't there be deletes coming in against those during the merge?
> IndexWriter.addIndexesNoOptimize(IndexReader[] readers)
> -------------------------------------------------------
>
> Key: LUCENE-1589
> URL: https://issues.apache.org/jira/browse/LUCENE-1589
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Similar to IndexWriter.addIndexesNoOptimize(Directory[] dirs)
> but for IndexReaders. This will be used to flush cloned ram
> indexes to disk for near realtime indexing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org