You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Simon Willnauer (JIRA)" <ji...@apache.org> on 2010/12/06 14:41:10 UTC
[jira] Created: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
DirectoryReader ignores NRT SegmentInfos in #isOptimized()
----------------------------------------------------------
Key: LUCENE-2802
URL: https://issues.apache.org/jira/browse/LUCENE-2802
Project: Lucene - Java
Issue Type: Bug
Components: Search
Affects Versions: 4.0
Reporter: Simon Willnauer
DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
{code}
public boolean isOptimized() {
ensureOpen();
// if segmentsInfos changes in IW this can return false positive
return segmentInfos.size() == 1 && !hasDeletions();
}
{code}
DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12967227#action_12967227 ]
Simon Willnauer commented on LUCENE-2802:
-----------------------------------------
bq. Nice catch Simon! This is also a thread safety issue since IR should not touch the writer's segmentInfos outside of sync(IW).
it seem like there is more about all that in DR - we should really only use the uncloned SegmentInfos if we are not in NRT mode #getVersion uses it too which is wrong.
I actually rely on the isOptimized in several tests and run into a NPE due to that though so we should really fix DR to use a private SegmentInfos or restrict the uncloned one for the isCurrent comparison
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Attachments: LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-2802:
------------------------------------
Affects Version/s: 3.1
we need to backport to 3.x too
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 3.1, 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Attachments: LUCENE-2802.patch, LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969139#action_12969139 ]
Simon Willnauer commented on LUCENE-2802:
-----------------------------------------
Committed revision 1043277.
I keep it open for backporting
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 3.1, 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Attachments: LUCENE-2802.patch, LUCENE-2802.patch, LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-2802:
------------------------------------
Attachment: LUCENE-2802.patch
here is a patch that removes the mutable state from DirectoryReader in the NRT case. The actual reason IMO why this has been introduced was that the RT reader returns true from #isCurrent() if the wirter was closed which is actually wrong since closing a writer changes the index and the reader should see that change.
I also added a testcase for is current to check the semantics
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Attachments: LUCENE-2802.patch, LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968503#action_12968503 ]
Earwin Burrfoot commented on LUCENE-2802:
-----------------------------------------
Patch looks cool.
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 3.1, 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Attachments: LUCENE-2802.patch, LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12967213#action_12967213 ]
Michael McCandless commented on LUCENE-2802:
--------------------------------------------
Nice catch Simon! This is also a thread safety issue since IR should not touch the writer's segmentInfos outside of sync(IW).
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Attachments: LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer reassigned LUCENE-2802:
---------------------------------------
Assignee: Simon Willnauer
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Attachments: LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-2802:
------------------------------------
Attachment: LUCENE-2802.patch
here is a final patch - i will commit later!
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 3.1, 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Attachments: LUCENE-2802.patch, LUCENE-2802.patch, LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-2802:
------------------------------------
Attachment: LUCENE-2802.patch
here is a patch
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Attachments: LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer resolved LUCENE-2802.
-------------------------------------
Resolution: Fixed
Fix Version/s: 4.0
3.1
Ported back to 3x in revision 1043418.
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 3.1, 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2802.patch, LUCENE-2802.patch, LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12967238#action_12967238 ]
Earwin Burrfoot commented on LUCENE-2802:
-----------------------------------------
Heh, I've [mentioned|https://issues.apache.org/jira/browse/LUCENE-2355?focusedCommentId=12874857&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12874857] this earlier in LUCENE-2355. :)
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Attachments: LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2802) DirectoryReader ignores NRT
SegmentInfos in #isOptimized()
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968418#action_12968418 ]
Simon Willnauer commented on LUCENE-2802:
-----------------------------------------
I changed DirectoryReader to use the cloned version of SegmentInfos instead of two of them inconsistently. The only failure I get is on TestIndexWriterReader line 105
{code}
r1.close();
writer.close();
assertTrue(r2.isCurrent());
{code}
where the writer is closed and afterwards it checks if the r2 reader still is the current one which failes since the writer.close() method changes the version of the SegmentInfos. In my opinion this is actually the semantics I would expect from #isCurrent(), the question is if we would want to change the semantics to return false from #isCurrent if the writer we used to obtain the reader from is closed.
I think we should consider it for consistency and simplicity though.
> DirectoryReader ignores NRT SegmentInfos in #isOptimized()
> ----------------------------------------------------------
>
> Key: LUCENE-2802
> URL: https://issues.apache.org/jira/browse/LUCENE-2802
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Attachments: LUCENE-2802.patch
>
>
> DirectoryReader only takes shared (with IW) SegmentInfos into account in DirectoryReader#isOptimized(). This can return true even if the actual realtime reader sees more than one segments.
> {code}
> public boolean isOptimized() {
> ensureOpen();
> // if segmentsInfos changes in IW this can return false positive
> return segmentInfos.size() == 1 && !hasDeletions();
> }
> {code}
> DirectoryReader should check if this reader has a non-nul segmentInfosStart and use that instead
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org