You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (Created) (JIRA)" <ji...@apache.org> on 2011/12/07 18:18:41 UTC
[jira] [Created] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Make PKIndexSplitter and MultiPassIndexSplitter work per segment
----------------------------------------------------------------
Key: LUCENE-3626
URL: https://issues.apache.org/jira/browse/LUCENE-3626
Project: Lucene - Java
Issue Type: Improvement
Components: modules/other
Affects Versions: 4.0
Reporter: Uwe Schindler
Assignee: Uwe Schindler
Fix For: 4.0
Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165274#comment-13165274 ]
Michael McCandless commented on LUCENE-3626:
--------------------------------------------
Shouldn't we just fix MultiDV to "handle" SortedSource? Ie, merge the terms & assign ords, across all subs, just like FieldCache does (using MultiTermsEnum) if you ask for DocTermsIndex on a SlowMRWrapper.
It's OK for this op to be costly -- SlowMRReader advertises this...
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Fix For: 4.0
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164539#comment-13164539 ]
Uwe Schindler commented on LUCENE-3626:
---------------------------------------
Correction: It could work, but it does not rectify the complexity.
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 4.0
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166992#comment-13166992 ]
Uwe Schindler commented on LUCENE-3626:
---------------------------------------
Committed trunk revision: 1212894
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3626.patch, LUCENE-3626.patch
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165327#comment-13165327 ]
Uwe Schindler commented on LUCENE-3626:
---------------------------------------
Mike, I agree with that! +1
Still we might think about making the index splitter work per segment. The PKIndexSpliter already works with a standard Lucene Filter (by default TermRangeFilter), so it could handle the segmentreaders directly.
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Fix For: 4.0
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Uwe Schindler (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler reassigned LUCENE-3626:
-------------------------------------
Assignee: (was: Uwe Schindler)
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Fix For: 4.0
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164696#comment-13164696 ]
Robert Muir commented on LUCENE-3626:
-------------------------------------
I'm not complaining, but you marked my issue as breaking this functionality? thats insanity!
Lets revisit the situation:
* As of yesterday, if you used one of these tools, *all* docvalues fields were *silently dropped* and merge succeeded (data loss)
* I fixed this in LUCENE-3623, but in testing exposed the fact that with the data loss fixed, if you had a sortedsource, you would get a non-obvious NullPointerException deep in the docvalues codec stack.
* Because of this LUCENE-3624 changes the NullPointerException to UnsupportedOperationException so that its clear that this isn't working, and why.
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Fix For: 4.0
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164538#comment-13164538 ]
Uwe Schindler commented on LUCENE-3626:
---------------------------------------
This only affects Lucene 4.0, as 3.x has no DocValues. The PKIndexSplitter per-segment variant could be backported to 3.x, but the MultiPassIndexSplitter working on absolute doc-Ids cannot handle that as AtomicReaderContext containing the docBase is not available in 3.x.
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 4.0
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3626:
----------------------------------
Attachment: LUCENE-3626.patch
New patch for the changes in trunk. It is ready to commit.
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Fix For: 4.0
>
> Attachments: LUCENE-3626.patch, LUCENE-3626.patch
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164687#comment-13164687 ]
Robert Muir commented on LUCENE-3626:
-------------------------------------
this issue is *not* broken by LUCENE-3624.
Prior to LUCENE-3624, you just got a NullPointerException instead of a UnsupportedOperationException!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 4.0
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164689#comment-13164689 ]
Uwe Schindler commented on LUCENE-3626:
---------------------------------------
Instead of complaining help to fix it. There was no better relation for it.
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 4.0
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Uwe Schindler (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler resolved LUCENE-3626.
-----------------------------------
Resolution: Fixed
Assignee: Uwe Schindler
Committed 3.x revision: 1212906
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3626.patch, LUCENE-3626.patch
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3626:
----------------------------------
Affects Version/s: 3.5
Fix Version/s: 3.6
This can be backported to 3.6
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3626.patch, LUCENE-3626.patch
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3626:
----------------------------------
Attachment: LUCENE-3626.patch
Patch that makes PKIndexSplitter and MultiPassIndexSplitter work per segment.
It also fixes the stupid MultiReader initialize method which prevents finalness of fields. It also makes MultiReader.starts[]/readerIndex() public to get the doc bases.
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Fix For: 4.0
>
> Attachments: LUCENE-3626.patch
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3626) Make PKIndexSplitter and
MultiPassIndexSplitter work per segment
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164698#comment-13164698 ]
Uwe Schindler commented on LUCENE-3626:
---------------------------------------
Change the relations if you like and find better ones, I am out of this issue now - unassigned myself and I will unassign from other Lucene issues now because of this stupidness. Goodbye.
> Make PKIndexSplitter and MultiPassIndexSplitter work per segment
> ----------------------------------------------------------------
>
> Key: LUCENE-3626
> URL: https://issues.apache.org/jira/browse/LUCENE-3626
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/other
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Fix For: 4.0
>
>
> Spinoff from LUCENE-3624: DocValuesw merger throws exception on IW.addIndexes(SlowMultiReaderWrapper) as string-index like docvalues cannot provide asSortedSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org