You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Earwin Burrfoot (JIRA)" <ji...@apache.org> on 2010/03/28 21:39:27 UTC
[jira] Created: (LUCENE-2355) Refactor
Directory/Multi/SegmentReader creation/reopening/cloning/closing
Refactor Directory/Multi/SegmentReader creation/reopening/cloning/closing
-------------------------------------------------------------------------
Key: LUCENE-2355
URL: https://issues.apache.org/jira/browse/LUCENE-2355
Project: Lucene - Java
Issue Type: Improvement
Reporter: Earwin Burrfoot
*Reader lifecycle evolved over time to become some heavily tangled mess. It's hard to understand what's going on there, it's even harder to add some fields/logic while ensuring that all possible code paths preserve these fields/interact with the logic properly. While some of said mess is justified by the task at hand, a big part is just badly done copypaste and can be removed.
I am currently refactoring this and intended to open an issue with a working patch, but the task winded up somewhat bigger than I expected, so I'm opening it earlier to track stuff encountered/changed/fixed.
The list is by no means exhaustive.
- an iteration to create SRs is copypasted several times, one of them (IW) with wrong iteration bound
- it is also overly complex and can be folded for create/reopen cases
- readers sent to IndexReaderWarmer are termindexless/docstoreless on some occasions
- it is possible to clone() your way to readwrite NRT reader
- IndexDeletionPolicy is not always preserved through clones/reopens
- cloned readers share CoreReaders and, consequently, updated termsIndex/docStores
- threadlocal versions of fieldsReader/termsVector are bound to SR, not CoreReaders and thus are recreated on clone/reopen
- double-initialization for some fields (someone got lost and did this to be sure I guess), stupid assert checks ( qwe = new(); assert qwe != null )
- SR is not always recreated when compound status of underlying segment changes
- deleting already deleted doc marks deletions dirty and rewrites them
- lots of synchronization is done around Reader, while it can be narrowed down to norms/deletions/whatever
I did some structural modifications:
- CompositeReader extracts common code from DirectoryReader and MultiReader (complete)
- ReadonlyDirectoryReader and ReadonlySegmentReader are dead, MutableD/SReaders are introduced and carry all modification logic/fields (DR complete, SR in progress)
- WriterBackedReader encapsulates NRT reader logic (complete)
- CoreReaders split into CoreReaders, DocStores, TermInfos. All of these are immutable and SR is cloned when you need to change its mode (in progress)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2355) Refactor
Directory/Multi/SegmentReader creation/reopening/cloning/closing
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855964#action_12855964 ]
Earwin Burrfoot commented on LUCENE-2355:
-----------------------------------------
- Norm holds a reference to papa-SegmentReader forever (not so deadly when SRs null themselves up on close, but it's worse with final fields and even now I can construct a case that leaks noticeable memory)
> Refactor Directory/Multi/SegmentReader creation/reopening/cloning/closing
> -------------------------------------------------------------------------
>
> Key: LUCENE-2355
> URL: https://issues.apache.org/jira/browse/LUCENE-2355
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Earwin Burrfoot
>
> *Reader lifecycle evolved over time to become some heavily tangled mess. It's hard to understand what's going on there, it's even harder to add some fields/logic while ensuring that all possible code paths preserve these fields/interact with the logic properly. While some of said mess is justified by the task at hand, a big part is just badly done copypaste and can be removed.
> I am currently refactoring this and intended to open an issue with a working patch, but the task winded up somewhat bigger than I expected, so I'm opening it earlier to track stuff encountered/changed/fixed.
> The list is by no means exhaustive.
> - an iteration to create SRs is copypasted several times, one of them (IW) with wrong iteration bound
> - it is also overly complex and can be folded for create/reopen cases
> - readers sent to IndexReaderWarmer are termindexless/docstoreless on some occasions
> - it is possible to clone() your way to readwrite NRT reader
> - IndexDeletionPolicy is not always preserved through clones/reopens
> - cloned readers share CoreReaders and, consequently, updated termsIndex/docStores
> - threadlocal versions of fieldsReader/termsVector are bound to SR, not CoreReaders and thus are recreated on clone/reopen
> - double-initialization for some fields (someone got lost and did this to be sure I guess), stupid assert checks ( qwe = new(); assert qwe != null )
> - SR is not always recreated when compound status of underlying segment changes
> - deleting already deleted doc marks deletions dirty and rewrites them
> - lots of synchronization is done around Reader, while it can be narrowed down to norms/deletions/whatever
> I did some structural modifications:
> - CompositeReader extracts common code from DirectoryReader and MultiReader (complete)
> - ReadonlyDirectoryReader and ReadonlySegmentReader are dead, MutableD/SReaders are introduced and carry all modification logic/fields (DR complete, SR in progress)
> - WriterBackedReader encapsulates NRT reader logic (complete)
> - CoreReaders split into CoreReaders, DocStores, TermInfos. All of these are immutable and SR is cloned when you need to change its mode (in progress)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2355) Refactor
Directory/Multi/SegmentReader creation/reopening/cloning/closing
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852417#action_12852417 ]
Earwin Burrfoot commented on LUCENE-2355:
-----------------------------------------
Some more stuff I forgot:
- No need to keep around origInstance in CoreReaders. It is used to purge FieldCache, but the cache is purged based on getFieldCacheKey, which is common across all clones/reopens
- MultiReader-level norms cache is not needed with per-segment searching and is removed. The method that extracts norms into a user-passed array still has the same performance even without the cache.
> Refactor Directory/Multi/SegmentReader creation/reopening/cloning/closing
> -------------------------------------------------------------------------
>
> Key: LUCENE-2355
> URL: https://issues.apache.org/jira/browse/LUCENE-2355
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Earwin Burrfoot
>
> *Reader lifecycle evolved over time to become some heavily tangled mess. It's hard to understand what's going on there, it's even harder to add some fields/logic while ensuring that all possible code paths preserve these fields/interact with the logic properly. While some of said mess is justified by the task at hand, a big part is just badly done copypaste and can be removed.
> I am currently refactoring this and intended to open an issue with a working patch, but the task winded up somewhat bigger than I expected, so I'm opening it earlier to track stuff encountered/changed/fixed.
> The list is by no means exhaustive.
> - an iteration to create SRs is copypasted several times, one of them (IW) with wrong iteration bound
> - it is also overly complex and can be folded for create/reopen cases
> - readers sent to IndexReaderWarmer are termindexless/docstoreless on some occasions
> - it is possible to clone() your way to readwrite NRT reader
> - IndexDeletionPolicy is not always preserved through clones/reopens
> - cloned readers share CoreReaders and, consequently, updated termsIndex/docStores
> - threadlocal versions of fieldsReader/termsVector are bound to SR, not CoreReaders and thus are recreated on clone/reopen
> - double-initialization for some fields (someone got lost and did this to be sure I guess), stupid assert checks ( qwe = new(); assert qwe != null )
> - SR is not always recreated when compound status of underlying segment changes
> - deleting already deleted doc marks deletions dirty and rewrites them
> - lots of synchronization is done around Reader, while it can be narrowed down to norms/deletions/whatever
> I did some structural modifications:
> - CompositeReader extracts common code from DirectoryReader and MultiReader (complete)
> - ReadonlyDirectoryReader and ReadonlySegmentReader are dead, MutableD/SReaders are introduced and carry all modification logic/fields (DR complete, SR in progress)
> - WriterBackedReader encapsulates NRT reader logic (complete)
> - CoreReaders split into CoreReaders, DocStores, TermInfos. All of these are immutable and SR is cloned when you need to change its mode (in progress)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2355) Refactor
Directory/Multi/SegmentReader creation/reopening/cloning/closing
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856509#action_12856509 ]
Earwin Burrfoot commented on LUCENE-2355:
-----------------------------------------
- Norms do both backward and forward sharing of their byte array. Eg, we clone a reader, then load up norms on the clone, and original sees and uses the same byte array.
This is really complex and likely noone ever needs it. Even if you do hit such a situation, and norms are loaded twice, first for a clone and then for original reader, it is transient and norms are really smallish compared to other memory hogs, so we can live with it.
I'm nixing this sharing. Successfully loaded norms are simply inherited from papa-reader.
> Refactor Directory/Multi/SegmentReader creation/reopening/cloning/closing
> -------------------------------------------------------------------------
>
> Key: LUCENE-2355
> URL: https://issues.apache.org/jira/browse/LUCENE-2355
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Earwin Burrfoot
>
> *Reader lifecycle evolved over time to become some heavily tangled mess. It's hard to understand what's going on there, it's even harder to add some fields/logic while ensuring that all possible code paths preserve these fields/interact with the logic properly. While some of said mess is justified by the task at hand, a big part is just badly done copypaste and can be removed.
> I am currently refactoring this and intended to open an issue with a working patch, but the task winded up somewhat bigger than I expected, so I'm opening it earlier to track stuff encountered/changed/fixed.
> The list is by no means exhaustive.
> - an iteration to create SRs is copypasted several times, one of them (IW) with wrong iteration bound
> - it is also overly complex and can be folded for create/reopen cases
> - readers sent to IndexReaderWarmer are termindexless/docstoreless on some occasions
> - it is possible to clone() your way to readwrite NRT reader
> - IndexDeletionPolicy is not always preserved through clones/reopens
> - cloned readers share CoreReaders and, consequently, updated termsIndex/docStores
> - threadlocal versions of fieldsReader/termsVector are bound to SR, not CoreReaders and thus are recreated on clone/reopen
> - double-initialization for some fields (someone got lost and did this to be sure I guess), stupid assert checks ( qwe = new(); assert qwe != null )
> - SR is not always recreated when compound status of underlying segment changes
> - deleting already deleted doc marks deletions dirty and rewrites them
> - lots of synchronization is done around Reader, while it can be narrowed down to norms/deletions/whatever
> I did some structural modifications:
> - CompositeReader extracts common code from DirectoryReader and MultiReader (complete)
> - ReadonlyDirectoryReader and ReadonlySegmentReader are dead, MutableD/SReaders are introduced and carry all modification logic/fields (DR complete, SR in progress)
> - WriterBackedReader encapsulates NRT reader logic (complete)
> - CoreReaders split into CoreReaders, DocStores, TermInfos. All of these are immutable and SR is cloned when you need to change its mode (in progress)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2355) Refactor
Directory/Multi/SegmentReader creation/reopening/cloning/closing
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853244#action_12853244 ]
Earwin Burrfoot commented on LUCENE-2355:
-----------------------------------------
- DR.clone() doesn't do ensureOpen(), you can successfully clone a DR over empty index, or get weird exception from underlying SR on non-empty index. If you relax synchronized blocks guarding clone/reopen for readonly readers (which is okay, in theory), it gets even more fun, as no checks are done when incRefing SR.CoreReaders.
I think I'm going to introduce interface RefCounted, abstract class WhateverRefCounted, which guards against increment on closed instance and has nice mass-decRef methods in IOUtils.
> Refactor Directory/Multi/SegmentReader creation/reopening/cloning/closing
> -------------------------------------------------------------------------
>
> Key: LUCENE-2355
> URL: https://issues.apache.org/jira/browse/LUCENE-2355
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Earwin Burrfoot
>
> *Reader lifecycle evolved over time to become some heavily tangled mess. It's hard to understand what's going on there, it's even harder to add some fields/logic while ensuring that all possible code paths preserve these fields/interact with the logic properly. While some of said mess is justified by the task at hand, a big part is just badly done copypaste and can be removed.
> I am currently refactoring this and intended to open an issue with a working patch, but the task winded up somewhat bigger than I expected, so I'm opening it earlier to track stuff encountered/changed/fixed.
> The list is by no means exhaustive.
> - an iteration to create SRs is copypasted several times, one of them (IW) with wrong iteration bound
> - it is also overly complex and can be folded for create/reopen cases
> - readers sent to IndexReaderWarmer are termindexless/docstoreless on some occasions
> - it is possible to clone() your way to readwrite NRT reader
> - IndexDeletionPolicy is not always preserved through clones/reopens
> - cloned readers share CoreReaders and, consequently, updated termsIndex/docStores
> - threadlocal versions of fieldsReader/termsVector are bound to SR, not CoreReaders and thus are recreated on clone/reopen
> - double-initialization for some fields (someone got lost and did this to be sure I guess), stupid assert checks ( qwe = new(); assert qwe != null )
> - SR is not always recreated when compound status of underlying segment changes
> - deleting already deleted doc marks deletions dirty and rewrites them
> - lots of synchronization is done around Reader, while it can be narrowed down to norms/deletions/whatever
> I did some structural modifications:
> - CompositeReader extracts common code from DirectoryReader and MultiReader (complete)
> - ReadonlyDirectoryReader and ReadonlySegmentReader are dead, MutableD/SReaders are introduced and carry all modification logic/fields (DR complete, SR in progress)
> - WriterBackedReader encapsulates NRT reader logic (complete)
> - CoreReaders split into CoreReaders, DocStores, TermInfos. All of these are immutable and SR is cloned when you need to change its mode (in progress)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2355) Refactor
Directory/Multi/SegmentReader creation/reopening/cloning/closing
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853280#action_12853280 ]
Earwin Burrfoot commented on LUCENE-2355:
-----------------------------------------
- when SegmentReader / CoreReaders are created, and exception is thrown, that exception can be hidden with exception originating in cleanup code (doClose/decRef)
- exception in doClose/decRef prevents closing and freeing all the resources (I believe I saw this once in the wild, when faulty merges ate up all file handles)
> Refactor Directory/Multi/SegmentReader creation/reopening/cloning/closing
> -------------------------------------------------------------------------
>
> Key: LUCENE-2355
> URL: https://issues.apache.org/jira/browse/LUCENE-2355
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Earwin Burrfoot
>
> *Reader lifecycle evolved over time to become some heavily tangled mess. It's hard to understand what's going on there, it's even harder to add some fields/logic while ensuring that all possible code paths preserve these fields/interact with the logic properly. While some of said mess is justified by the task at hand, a big part is just badly done copypaste and can be removed.
> I am currently refactoring this and intended to open an issue with a working patch, but the task winded up somewhat bigger than I expected, so I'm opening it earlier to track stuff encountered/changed/fixed.
> The list is by no means exhaustive.
> - an iteration to create SRs is copypasted several times, one of them (IW) with wrong iteration bound
> - it is also overly complex and can be folded for create/reopen cases
> - readers sent to IndexReaderWarmer are termindexless/docstoreless on some occasions
> - it is possible to clone() your way to readwrite NRT reader
> - IndexDeletionPolicy is not always preserved through clones/reopens
> - cloned readers share CoreReaders and, consequently, updated termsIndex/docStores
> - threadlocal versions of fieldsReader/termsVector are bound to SR, not CoreReaders and thus are recreated on clone/reopen
> - double-initialization for some fields (someone got lost and did this to be sure I guess), stupid assert checks ( qwe = new(); assert qwe != null )
> - SR is not always recreated when compound status of underlying segment changes
> - deleting already deleted doc marks deletions dirty and rewrites them
> - lots of synchronization is done around Reader, while it can be narrowed down to norms/deletions/whatever
> I did some structural modifications:
> - CompositeReader extracts common code from DirectoryReader and MultiReader (complete)
> - ReadonlyDirectoryReader and ReadonlySegmentReader are dead, MutableD/SReaders are introduced and carry all modification logic/fields (DR complete, SR in progress)
> - WriterBackedReader encapsulates NRT reader logic (complete)
> - CoreReaders split into CoreReaders, DocStores, TermInfos. All of these are immutable and SR is cloned when you need to change its mode (in progress)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org