You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (Created) (JIRA)" <ji...@apache.org> on 2011/12/13 13:55:30 UTC
[jira] [Created] (LUCENE-3644) problems with IR's
readerFinishedListener
problems with IR's readerFinishedListener
-----------------------------------------
Key: LUCENE-3644
URL: https://issues.apache.org/jira/browse/LUCENE-3644
Project: Lucene - Java
Issue Type: Bug
Reporter: Robert Muir
Fix For: 3.6, 4.0
There are two major problems:
1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
We should discuss how to fix #1.
I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Robert Muir (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir reassigned LUCENE-3644:
-----------------------------------
Assignee: Robert Muir
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3644.patch, LUCENE-3644.patch, LUCENE-3644.patch
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168396#comment-13168396 ]
Uwe Schindler commented on LUCENE-3644:
---------------------------------------
And "ReaderFinished" is wrong word, it is "ReaderClosed". Finished sounds more like "Ready" than like "Close", its misunderstanding. I heard of that shit the first time last weekend and was teriified. Who proposed it, I will kill him/her?
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168391#comment-13168391 ]
Robert Muir commented on LUCENE-3644:
-------------------------------------
{quote}
That's a good idea: separate APIs for "core closed event" and "composite reader closed event".
{quote}
Thats not really what i'm saying though.
I think readerFinishedListener, since its on indexreader, should fire when an IndexReader closes.
I dont care if its composite or not. Its an indexreader, thats all we care about.
Then core closed event naturally makes sense on SegmentReader, because its the only one that
cares/knows about cores.
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3644:
--------------------------------
Attachment: LUCENE-3644.patch
attached is a patch, cleaning this up. All tests pass.
The two use cases are now totally separated instead of being confusingly and dangerously interwoven. I also renamed things per uwe's advice:
* IndexReader's ReaderClosedListener does exactly what it says. It fires when readers close. This is used to e.g. evict top-level fieldcaches.
* SegmentReader's CoreClosedListener fires when the shared core is closed. This is used to e.g. evict per-segment fieldcaches.
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3644.patch
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168373#comment-13168373 ]
Michael McCandless commented on LUCENE-3644:
--------------------------------------------
The original intention of this API (LUCENE-2474) was in fact to detect
only when a SegmentCoreReader was closed, so that "external" caches
the app kept could be purged, and not really to detect closing of all
other readers (eg composite readers).
But I agree, we should somehow generalize this to do both
functions...
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168393#comment-13168393 ]
Uwe Schindler commented on LUCENE-3644:
---------------------------------------
I hate the whole thing, sorry. Please remove it,... completely.
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Robert Muir (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-3644.
---------------------------------
Resolution: Fixed
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3644.patch, LUCENE-3644.patch, LUCENE-3644.patch
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3644:
--------------------------------
Attachment: LUCENE-3644.patch
updated patch, with a stupid optimization for when you have field cache on top-level readers and also insanity.
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3644.patch, LUCENE-3644.patch
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168643#comment-13168643 ]
Michael McCandless commented on LUCENE-3644:
--------------------------------------------
+1 looks great!
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3644.patch, LUCENE-3644.patch, LUCENE-3644.patch
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Shay Banon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182082#comment-13182082 ]
Shay Banon commented on LUCENE-3644:
------------------------------------
I just saw this thread, and would like to respond to @Uwe as I think he is completely wrong here. Its perfectly fine to have listeners in this case. If you are implementing a cache based on core readers (think custom filter cache for example), its basically the only sane way to do it (and not have to handle weak references which is bad on GC and error prone with complex keys). And Lucene should allow for that.
I don't think the separation to segment core listener and index reader listener is the best solution here, since it exposes "too much info" to the user (having to deal with different reader cases, check FieldCacheImpl for example). I don't really mind it, since in my case I only work on low level Segment case (which is the sane way to work when you do NRT) and can handle it, but not sure its best in a general usability wise case. What the user want is: here is a reader, which I case base on it, and here is a listener, call me back when that listener is no longer valid.
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3644.patch, LUCENE-3644.patch, LUCENE-3644.patch
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168482#comment-13168482 ]
Uwe Schindler commented on LUCENE-3644:
---------------------------------------
Hi, I am happy with the code now, but I still hate the (Event-)Listener-pattern. In a library like Lucene its very "foreign", it fits much better to GUIs with event loops :-)
And thanks for renaming the Listener!
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3644.patch, LUCENE-3644.patch
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Shay Banon (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182082#comment-13182082 ]
Shay Banon edited comment on LUCENE-3644 at 1/7/12 8:22 PM:
------------------------------------------------------------
I just saw this thread, and would like to respond to @Uwe as I think he is completely wrong here. Its perfectly fine to have listeners in this case. If you are implementing a cache based on core readers (think custom filter cache for example), its basically the only sane way to do it (and not have to handle weak references which is bad on GC and error prone with complex keys). And Lucene should allow for that.
I don't think the separation to segment core listener and index reader listener is the best solution here, since it exposes "too much info" to the user (having to deal with different reader cases, check FieldCacheImpl for example). I don't really mind it, since in my case I only work on low level Segment case (which is the sane way to work when you do NRT) and can handle it, but not sure its best in a general usability wise case. What the user want is: here is a reader, which I cache base on it, and here is a listener, call me back when that listener is no longer valid.
was (Author: kimchy):
I just saw this thread, and would like to respond to @Uwe as I think he is completely wrong here. Its perfectly fine to have listeners in this case. If you are implementing a cache based on core readers (think custom filter cache for example), its basically the only sane way to do it (and not have to handle weak references which is bad on GC and error prone with complex keys). And Lucene should allow for that.
I don't think the separation to segment core listener and index reader listener is the best solution here, since it exposes "too much info" to the user (having to deal with different reader cases, check FieldCacheImpl for example). I don't really mind it, since in my case I only work on low level Segment case (which is the sane way to work when you do NRT) and can handle it, but not sure its best in a general usability wise case. What the user want is: here is a reader, which I case base on it, and here is a listener, call me back when that listener is no longer valid.
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3644.patch, LUCENE-3644.patch, LUCENE-3644.patch
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168351#comment-13168351 ]
Uwe Schindler commented on LUCENE-3644:
---------------------------------------
Hi,
+1, I was f*cking with this the whole day when rewriting oirectoryReader/MultiReader.
The bad thing is also that the Set(CHM) is not initialized in IndexReader base class (and is not final), the subclasses must do it. This should be done by base class and be protected final, then normal IndexReaders dont need to f*ck with it. Just make the listeners collection in IndexReader base class final and initialize with MapBackedSet(CHM).
DirectoryReader and other MultiReaders may simply delegate the registration to the subreaders, but dont pass sets around. If they need to copy all, just do "for (Listener l: this.listenerss) subreader[i].add(l);", sharing the same map between different readers is wrong.
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168378#comment-13168378 ]
Robert Muir commented on LUCENE-3644:
-------------------------------------
I think thats confusing, I think if you set the listener on a reader, it should work.
If we want a *core* listener, it should be a separate thing! and we can have a clean API for that,
specific to SR (Maybe even only on SR itself!) that doesn't confuse the other case
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168398#comment-13168398 ]
Uwe Schindler commented on LUCENE-3644:
---------------------------------------
Sorry for the last comment, I am still in rage, rage against this... aaaaah ! :( die, die, die!
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3644:
--------------------------------
Attachment: LUCENE-3644.patch
{quote}
But: I don't think we should "forward to subs" when someone registers for reader closed? I think if you register you should only be notified when that exact reader is closed?
{quote}
I agree, the only two real uses of this api we have today (as of the previous patch), both jump thru hoops to avoid this. Here's a patch implementing it that way (much cleaner imo)
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3644.patch, LUCENE-3644.patch, LUCENE-3644.patch
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168475#comment-13168475 ]
Michael McCandless commented on LUCENE-3644:
--------------------------------------------
Patch looks great! I love the separation into "reader closed" vs "core closed", and removing all that hairy code that was copying around the previously registered listeners... MUCH simpler.
But: I don't think we should "forward to subs" when someone registers for reader closed? I think if you register you should only be notified when that exact reader is closed? If somehow you care about the subs (I think this is the exception not the rule) you can always pull them and register with them too.
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3644.patch, LUCENE-3644.patch
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3644) problems with IR's
readerFinishedListener
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168386#comment-13168386 ]
Michael McCandless commented on LUCENE-3644:
--------------------------------------------
That's a good idea: separate APIs for "core closed event" and "composite reader closed event".
> problems with IR's readerFinishedListener
> -----------------------------------------
>
> Key: LUCENE-3644
> URL: https://issues.apache.org/jira/browse/LUCENE-3644
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
>
> There are two major problems:
> 1. The listener api does not really apply all indexreaders. for example segmentreaders dont fire it on close, only segmentcorereaders. this is wrong, a segmentcorereader is *not* an indexreader. Furthermore, if you register it on a top-level reader you get events for anything under the reader tree (sometimes, unless they are segmentreaders as mentioned above, where it doesnt work correctly at all).
> 2. Furthermore your listener is 'passed along' in a viral fashion from clone() and reopen(). This means for example, if you are trying to listen to readers in NRT search you are just accumulating reader listeners, all potentially keeping references to old indexreaders (because, in order to deal with #1 your listener must 'keep' a reference to the IR it was registered on, so it can check if thats *really* the one).
> We should discuss how to fix #1.
> I will create a patch for #2 shortly and commit it, its just plain wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org