You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (Created) (JIRA)" <ji...@apache.org> on 2011/12/09 12:54:40 UTC
[jira] [Created] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
-------------------------------------------------------------------------------------------------------------
Key: LUCENE-3631
URL: https://issues.apache.org/jira/browse/LUCENE-3631
Project: Lucene - Java
Issue Type: Task
Components: core/index
Affects Versions: 4.0
Reporter: Uwe Schindler
After LUCENE-3606 is finished, there are some TODOs:
SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
There are two possibilities to do this:
# the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
# Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227042#comment-13227042 ]
Michael McCandless commented on LUCENE-3631:
--------------------------------------------
bq. Guys, is it possible in a current implementation to update the doc values fields without re-indexing a whole document?
Not yet, though LUCENE-3837 is exploring how to update postings (not doc values though) for one field in a doc...
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631-threadlocals.patch, LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173975#comment-13173975 ]
Simon Willnauer commented on LUCENE-3631:
-----------------------------------------
bq. One thing: we have thread locals for TermVectorsReader and StoredFieldsReader. Would it make sense to use one for DocValues, too? What do you think Simon?
I don't see a need for this. The source is cached in the DocValues instance and DocValues instances can be shared across thread.
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172266#comment-13172266 ]
Michael McCandless commented on LUCENE-3631:
--------------------------------------------
bq. Why not remove the Cloneable interface from IndexReader and remove clone alltogether. Then user gets compile error.
Ahh, right! I'll do that.
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172521#comment-13172521 ]
Michael McCandless commented on LUCENE-3631:
--------------------------------------------
Oh that's silly... I'll fix before committing.
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Michael McCandless (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3631:
---------------------------------------
Attachment: LUCENE-3631.patch
Patch, making SegmentReader fully read-only; it's much simpler now! I
cutover to simple ctors (3 of them, depending on how the core/del docs
are set) instead of static get methods.
I removed clone() from all IR impls and made final IR.clone() that throws
UOE. Cloning no longer makes sense since all IRs are read-only...
I pushed all "pending deletes" tracking inside IW, and changed the
readerPool to store a map of SegmentInfo to the "state" that IW needs
to track for each segment (open reader for searching, open reader for
merging, live docs and pending delete count).
I think it's ready...
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172353#comment-13172353 ]
Michael McCandless commented on LUCENE-3631:
--------------------------------------------
Hmm I meant to "svn rm TestIndexReaderClone.java".
Oh I'll remove that silly sop.
You're right -- we don't need any sync in SegmentCoreReaders now! It's entirely final (also SegmentReader)... I'll remove.
{quote}
And in my opinion incRef/decRef on IndexReaders is useless, too (or almost useless, except segmentreaders). As Shai said before, code like SearcherManager should reCount externaly with a separate interface/helperclass (that refcounts and on decRef to 0, it calls Closeable.close - so helper can work on any Closeable). But we should remove that in separate issue.
{quote}
We should explore that... it's tricky.
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172309#comment-13172309 ]
Uwe Schindler commented on LUCENE-3631:
---------------------------------------
I get a clone error in TestIndexReaderClone. There s also still a strange verbosity:
{noformat}
[junit] COMPUTE TEST METHODS: org.apache.lucene.util.LuceneTestCaseRunner@27f649df
{noformat}
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3631:
----------------------------------
Attachment: LUCENE-3631-threadlocals.patch
This patch also moves the threadlocals to SegmentCoreReaders, as they can be reused on reopen/nrt readers. Also improve ensureOpen() checks to guard everything without duplicating checks.
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631-threadlocals.patch, LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173959#comment-13173959 ]
Uwe Schindler commented on LUCENE-3631:
---------------------------------------
Hi,
I committed some small cleanups and dead code removal after Clover analysis this morning.
One thing: we have thread locals for TermVectorsReader and StoredFieldsReader. Would it make sense to use one for DocValues, too? What do you think Simon?
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174012#comment-13174012 ]
Uwe Schindler commented on LUCENE-3631:
---------------------------------------
Heavy committed at revision: 1221677
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631-threadlocals.patch, LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172526#comment-13172526 ]
Robert Muir commented on LUCENE-3631:
-------------------------------------
+1
I think we should look at moving deletes to codec after you commit.
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172514#comment-13172514 ]
Uwe Schindler commented on LUCENE-3631:
---------------------------------------
Looks good! I think thats a good improvement!
One very small thing, can also be fixed later, no need for new patch: SegmentReader.toString() creates a StringBuilder, appends *one* String to it and returns it with toString() :-) Maybe simply retun the SegmentInfo.toString(...) result...
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Michael McCandless (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3631:
---------------------------------------
Attachment: LUCENE-3631.patch
New patch with Uwe's suggestions.. I think it's ready!
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Aliaksandr Zhuhrou (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225210#comment-13225210 ]
Aliaksandr Zhuhrou commented on LUCENE-3631:
--------------------------------------------
Guys, is it possible in a current implementation to update the doc values fields without re-indexing a whole document?
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631-threadlocals.patch, LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Michael McCandless (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reassigned LUCENE-3631:
------------------------------------------
Assignee: Michael McCandless
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172284#comment-13172284 ]
Uwe Schindler commented on LUCENE-3631:
---------------------------------------
Patch looks nice.
One more thing: In my opinion, the whole synchronized stuff in SegmentCoreReaders can be removed (see LUCENE-3653) - all is final now, why synchronize? So LUCENE-3653 is a non-issue, when this is committed. Same applies to IR/DR/... - almost all synchronized methods can go away.
And in my opinion incRef/decRef on IndexReaders is useless, too (or almost useless, except segmentreaders). As Shai said before, code like SearcherManager should reCount externaly with a separate interface/helperclass (that refcounts and on decRef to 0, it calls Closeable.close - so helper can work on any Closeable). But we should remove that in future.
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173983#comment-13173983 ]
Uwe Schindler commented on LUCENE-3631:
---------------------------------------
bq. The source is cached in the DocValues instance and DocValues instances can be shared across thread.
Thanks, I just wanted to make sure that there is no synchronization on DocValues. A customer of mine had huge improvements with loading stored fields since this is in Lucene.
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Aliaksandr Zhuhrou (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229687#comment-13229687 ]
Aliaksandr Zhuhrou commented on LUCENE-3631:
--------------------------------------------
Yes. I thought on something like that. I see that lucene enforces a write-once policy (as I understand this needed to support a transactional behaviour). So I think we may use some conbination of approach for the LUCENE-3837 and a current segment system where any updates after commit go to the a new file like we do with segments. And when the docValues will be read to a memory we will fetch all files and values in a file with higher generation will override values with same docIds.
Of course it increases even more the total amount of files. And not sure yet how to express this updates on level of the IndexWriter api.
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631-threadlocals.patch, LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172212#comment-13172212 ]
Uwe Schindler commented on LUCENE-3631:
---------------------------------------
Why not remove the Cloneable interface from IndexReader and remove clone alltogether. Then user gets compile error.
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3631) Remove write access
from SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Uwe Schindler (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172284#comment-13172284 ]
Uwe Schindler edited comment on LUCENE-3631 at 12/19/11 2:01 PM:
-----------------------------------------------------------------
Patch looks nice.
One more thing: In my opinion, the whole synchronized stuff in SegmentCoreReaders can be removed (see LUCENE-3653) - all is final now, why synchronize? So LUCENE-3653 is a non-issue, when this is committed. Same applies to IR/DR/... - almost all synchronized methods can go away. And decRef, too, as it's already guarded by AtomicRef
And in my opinion incRef/decRef on IndexReaders is useless, too (or almost useless, except segmentreaders). As Shai said before, code like SearcherManager should reCount externaly with a separate interface/helperclass (that refcounts and on decRef to 0, it calls Closeable.close - so helper can work on any Closeable). But we should remove that in separate issue.
was (Author: thetaphi):
Patch looks nice.
One more thing: In my opinion, the whole synchronized stuff in SegmentCoreReaders can be removed (see LUCENE-3653) - all is final now, why synchronize? So LUCENE-3653 is a non-issue, when this is committed. Same applies to IR/DR/... - almost all synchronized methods can go away.
And in my opinion incRef/decRef on IndexReaders is useless, too (or almost useless, except segmentreaders). As Shai said before, code like SearcherManager should reCount externaly with a separate interface/helperclass (that refcounts and on decRef to 0, it calls Closeable.close - so helper can work on any Closeable). But we should remove that in future.
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3631) Remove write access from
SegmentReader and possibly move to separate class or
IndexWriter/BufferedDeletes/...
Posted by "Michael McCandless (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-3631.
----------------------------------------
Resolution: Fixed
> Remove write access from SegmentReader and possibly move to separate class or IndexWriter/BufferedDeletes/...
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3631
> URL: https://issues.apache.org/jira/browse/LUCENE-3631
> Project: Lucene - Java
> Issue Type: Task
> Components: core/index
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Attachments: LUCENE-3631.patch, LUCENE-3631.patch
>
>
> After LUCENE-3606 is finished, there are some TODOs:
> SegmentReader still contains (package-private) all delete logic including crazy copyOnWrite for validDocs Bits. It would be good, if SegmentReader itsself could be read-only like all other IndexReaders.
> There are two possibilities to do this:
> # the simple one: Subclass SegmentReader and make a RWSegmentReader that is only used by IndexWriter/BufferedDeletes/... DirectoryReader will only use the read-only SegmentReader. This would move all TODOs to a separate class. It's reopen/clone method would always create a RO-SegmentReader (for NRT).
> # Remove all write and commit stuff from SegmentReader completely and move it to IndexWriter's readerPool (it must be in readerPool as deletions need a not-changing view on an index snapshot).
> Unfortunately the code is so complicated and I have no real experience in those internals of IndexWriter so I did not want to do it with LUCENE-3606, I just separated the code in SegmentReader and marked with TODO. Maybe Mike McCandless can help :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org