You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Jason Rutherglen (JIRA)" <ji...@apache.org> on 2009/11/16 20:01:39 UTC

[jira] Created: (LUCENE-2071) Allow upating of IndexWriter SegmentReaders

Allow upating of IndexWriter SegmentReaders
-------------------------------------------

                 Key: LUCENE-2071
                 URL: https://issues.apache.org/jira/browse/LUCENE-2071
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Index
    Affects Versions: 2.9.1
            Reporter: Jason Rutherglen
            Priority: Minor
             Fix For: 3.1


This discussion kind of started in LUCENE-2047.  Basically, we'll allow users to perform delete document, and norms updates on SegmentReaders that are handled by IndexWriter.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Updated: (LUCENE-2071) Allow updating of IndexWriter SegmentReaders

Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-2071:
-------------------------------------

    Summary: Allow updating of IndexWriter SegmentReaders  (was: Allow upating of IndexWriter SegmentReaders)

> Allow updating of IndexWriter SegmentReaders
> --------------------------------------------
>
>                 Key: LUCENE-2071
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2071
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.9.1
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2071.patch
>
>
> This discussion kind of started in LUCENE-2047.  Basically, we'll allow users to perform delete document, and norms updates on SegmentReaders that are handled by IndexWriter.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2071) Allow updating of IndexWriter SegmentReaders

Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778624#action_12778624 ] 

Michael McCandless commented on LUCENE-2071:
--------------------------------------------

I would rather not open up such an advanced API now.  What's the
specific driver (that can't be achieved throw IW's already exposed
APIs)?

Or, at least, take this up in the larger context of the refactoring of
IW / allowing custom subclasses of SR to be used by apps / etc.

> Allow updating of IndexWriter SegmentReaders
> --------------------------------------------
>
>                 Key: LUCENE-2071
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2071
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.9.1
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2071.patch
>
>
> This discussion kind of started in LUCENE-2047.  Basically, we'll allow users to perform delete document, and norms updates on SegmentReaders that are handled by IndexWriter.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2071) Allow updating of IndexWriter SegmentReaders

Posted by "Tim Smith (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851388#action_12851388 ] 

Tim Smith commented on LUCENE-2071:
-----------------------------------

+1

I have a special subclassed IndexSearcher that certain special queries require, so IndexWriter's delete by query will fail as an IndexSearcher is passed in

With this added method, i would be able to construct my own Searcher over the readers and then apply deletes properly

This would also allow counting the deletes as they occur as well (which is commonly desired when deleting by query)


It would be nice if this method would also work with non-pooled readers

so my desired method signature would be:
void updateReaders(Readers callback, boolean pooled)

if the readers were already pooled, this would have no effect, otherwise it would just open the segment readers just like the non-pooled delete readers are opened

> Allow updating of IndexWriter SegmentReaders
> --------------------------------------------
>
>                 Key: LUCENE-2071
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2071
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.9.1
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2071.patch
>
>
> This discussion kind of started in LUCENE-2047.  Basically, we'll allow users to perform delete document, and norms updates on SegmentReaders that are handled by IndexWriter.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2071) Allow updating of IndexWriter SegmentReaders

Posted by "Tim Smith (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851528#action_12851528 ] 

Tim Smith commented on LUCENE-2071:
-----------------------------------

found a couple of small issues with the patch attached to this ticket:

1. applyDeletes issue

saw this was in another ticket

think the flush should be flush(true, true, false)
and applyDeletes() should be called in the synchronized block


2. IndexWriter.changeCount not updated

the call() method does not return a boolean indicating if there were any changes that would need to be committed

as a result, if no other changes are made to the indexwriter, the commit will be skipped, even though deletes/norm updates were sent in
IndexReader.reopen() will then return the "old" reader without the deletes/norms



> Allow updating of IndexWriter SegmentReaders
> --------------------------------------------
>
>                 Key: LUCENE-2071
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2071
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.9.1
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2071.patch
>
>
> This discussion kind of started in LUCENE-2047.  Basically, we'll allow users to perform delete document, and norms updates on SegmentReaders that are handled by IndexWriter.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2071) Allow updating of IndexWriter SegmentReaders

Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778641#action_12778641 ] 

Jason Rutherglen commented on LUCENE-2071:
------------------------------------------

I suspect there's apps out in the wild that could use this (I
recall some of the LinkedIn code could find this useful,
something like Zoie that already has deletes calculated in a doc
set, and I doubt it's the only installation with this use case).
Solr has pre-calculated docsets which could operate on exposed
SRs as well. If we offer this sooner rather than later, then we
needn't worry about users writing crazy workarounds later (i.e.
popping docsets into a filter query) just to perform deletes
when using Lucene NRT.

I'd rather not limit people's using NRT/getReader because they rely
on per doc deletes, which as it turns out is really trivial to
support (when LUCENE-1516 was written, I wasn't fully aware of
how deletes work, now that I know it's all synced on IW, I don't
think it's a big deal). 

I guess we should map out what the worst case scenarios are
(i.e. what is the worst thing the user can do with the mutable
SR access?). Then we can specifically limit it?

I'm not sure how the worst case scenario will change when we
refactor the IW APIs? Meaning the user will still need to be
advanced (probably more advanced after offering pluggable IW
APIs). They could still break things (unless we limit what's
breakable), and we'd still basically be offering SR access?

> Allow updating of IndexWriter SegmentReaders
> --------------------------------------------
>
>                 Key: LUCENE-2071
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2071
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.9.1
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2071.patch
>
>
> This discussion kind of started in LUCENE-2047.  Basically, we'll allow users to perform delete document, and norms updates on SegmentReaders that are handled by IndexWriter.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Updated: (LUCENE-2071) Allow upating of IndexWriter SegmentReaders

Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-2071:
-------------------------------------

    Attachment: LUCENE-2071.patch

Added an IW updateReaders method that accepts a Readers callback
class. The current segment readers are passed into the callback
and may be updated safely because we're in a sync block. The
reader pooling is transparent to the user.

> Allow upating of IndexWriter SegmentReaders
> -------------------------------------------
>
>                 Key: LUCENE-2071
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2071
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.9.1
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2071.patch
>
>
> This discussion kind of started in LUCENE-2047.  Basically, we'll allow users to perform delete document, and norms updates on SegmentReaders that are handled by IndexWriter.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org