You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Alan Woodward (JIRA)" <ji...@apache.org> on 2012/10/12 16:33:06 UTC
[jira] [Created] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Alan Woodward created LUCENE-4479:
-------------------------------------
Summary: TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
Key: LUCENE-4479
URL: https://issues.apache.org/jira/browse/LUCENE-4479
Project: Lucene - Core
Issue Type: Bug
Components: modules/highlighter
Affects Versions: 4.0
Reporter: Alan Woodward
Priority: Minor
The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
"Low level api. Returns a token stream or null if no offset info available
in index. This can be used to feed the highlighter with a pre-parsed token
stream"
However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475148#comment-13475148 ]
Robert Muir commented on LUCENE-4479:
-------------------------------------
Alan this patch looks good. Thanks for tackling this.
Do you know off-hand if we have a test anywhere for the case of offsets but NO positions (which there is logic to handle, and looks correct, I'm just curious).
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch, LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Alan Woodward (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Woodward resolved LUCENE-4479.
-----------------------------------
Resolution: Fixed
trunk: 1400504
branch_4x: 1400505
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Assignee: Alan Woodward
> Priority: Minor
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4479.patch, LUCENE-4479.patch, LUCENE-4479.patch, LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475045#comment-13475045 ]
Robert Muir commented on LUCENE-4479:
-------------------------------------
I'm not sure whats going on here :) If you can sort it out it would be fantastic.
In general if the method requires that the Terms object has offsets or positions the checks should just look like:
if (tpv.hasOffsets())
...
if (tpv.hasPositions())
...
But I'm really confused what the contracts of these methods should be. definitely bugs :)
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Alan Woodward (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475043#comment-13475043 ]
Alan Woodward commented on LUCENE-4479:
---------------------------------------
So this snippet:
if (!tokenPositionsGuaranteedContiguous && hasPositions(tpv)) {
return new TokenStreamFromTermPositionVector(tpv);
}
should presumably also check for offsets, and then the extra hasOffsets check in TSFTPV can be removed.
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir reassigned LUCENE-4479:
-----------------------------------
Assignee: Alan Woodward
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Assignee: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch, LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475039#comment-13475039 ]
Robert Muir commented on LUCENE-4479:
-------------------------------------
but the behavior of throwing IAE is still bogus, as it disagrees with the javadocs... I just feel like there is another bug involved too.
The javadocs are also wrong, because looking at the code logic it really needs positions AND offsets.
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Alan Woodward (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Woodward updated LUCENE-4479:
----------------------------------
Attachment: LUCENE-4479.patch
New patch, with CHANGES entry. Could someone cast a quick eye over it, and I'll commit.
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Assignee: Alan Woodward
> Priority: Minor
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4479.patch, LUCENE-4479.patch, LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Alan Woodward (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475155#comment-13475155 ]
Alan Woodward commented on LUCENE-4479:
---------------------------------------
TokenSourcesTest.testOverlapWithOffset() tests it implicitly, although it would be nice to set the FieldType explicitly to document it I suppose.
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch, LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475171#comment-13475171 ]
Robert Muir commented on LUCENE-4479:
-------------------------------------
you're right: thanks!
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch, LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Alan Woodward (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Woodward updated LUCENE-4479:
----------------------------------
Attachment: LUCENE-4479.patch
Patch with a test illustrating the problem
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Alan Woodward (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475055#comment-13475055 ]
Alan Woodward commented on LUCENE-4479:
---------------------------------------
It looks as though these classes are only used by the DefaultSolrHighlighter, which is broken under these circumstances. I'll make those changes, and see if any tests fail...
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Alan Woodward (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Woodward updated LUCENE-4479:
----------------------------------
Fix Version/s: 5.0
4.1
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Assignee: Alan Woodward
> Priority: Minor
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4479.patch, LUCENE-4479.patch, LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477990#comment-13477990 ]
Robert Muir commented on LUCENE-4479:
-------------------------------------
with the current patch, 'ant precommit' fails:
{noformat}
...
-ecj-javadoc-lint-src:
[ecj-lint] Compiling 45 source files
[ecj-lint] ----------
[ecj-lint] 1. ERROR in /Users/rmuir/workspace/lucene-trunk/lucene/highlighter/src/java/org/apache/lucene/search/highlight/TokenSources.java (at line 275)
[ecj-lint] * @throws IOException
[ecj-lint] ^^^^^^^^^^^
[ecj-lint] Javadoc: Description expected after this reference
[ecj-lint] ----------
[ecj-lint] 1 problem (1 error)
BUILD FAILED
{noformat}
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Assignee: Alan Woodward
> Priority: Minor
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4479.patch, LUCENE-4479.patch, LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475037#comment-13475037 ]
Robert Muir commented on LUCENE-4479:
-------------------------------------
{quote}
However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
{quote}
Actually thats not true: you will get an IllegalArgumentException because OffsetAttribute won't allow offsets to be set to negative values (and the DocsAndPositionsEnum will return -1).
So something else is going on here!
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Alan Woodward (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475041#comment-13475041 ]
Alan Woodward commented on LUCENE-4479:
---------------------------------------
Ah, but TokenStreamFromTermPositionVector checks if offsets are there, and works round it if they're not. Which is presumably also a bug?
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Alan Woodward (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Woodward updated LUCENE-4479:
----------------------------------
Attachment: LUCENE-4479.patch
Patch, fixing javadocs errors. Ant precommit runs successfully.
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Assignee: Alan Woodward
> Priority: Minor
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4479.patch, LUCENE-4479.patch, LUCENE-4479.patch, LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4479) TokenSources.getTokenStream()
doesn't return correctly for termvectors with positions but no offsets
Posted by "Alan Woodward (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Woodward updated LUCENE-4479:
----------------------------------
Attachment: LUCENE-4479.patch
Patch, changing the API slightly:
- getTokenStream(Terms, bool) now throws IllegalArgumentException if the Terms does not have offsets.
- renames getTokenStream(IndexReader, int, String) to getTokenStreamWithOffsets. Not sure I like the name... Also added some javadocs here. Instead of throwing IllegalArgumentExceptions, this just returns null if it can't build a tokenstream for whatever reason (no termvector, no offsets or positions).
The only API consumer here is DefaultSolrHighlighter, which I've edited accordingly. I also added a test to Solr to ensure that fields with termvectors and positions but no offsets are correctly highlighted.
All tests pass
> TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4479
> URL: https://issues.apache.org/jira/browse/LUCENE-4479
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Affects Versions: 4.0
> Reporter: Alan Woodward
> Priority: Minor
> Attachments: LUCENE-4479.patch, LUCENE-4479.patch
>
>
> The javadocs for TokenSources.getTokenStream(Terms, boolean) state:
> "Low level api. Returns a token stream or null if no offset info available
> in index. This can be used to feed the highlighter with a pre-parsed token
> stream"
> However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.
> This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org