Speeding up highlighting by storing a cached TokenStream

[Michael Stoppelman <st...@gmail.com>]

Most of the time the highlighting uses is in getting the next token from the
analyzer (tokenStream.next()). I'm wondering how I can access the tokens
that
are stored in lucene (or store another copy of the TokenStream seperately)
and send a pre-tokenized TokenStream to the highlighter so next() is speedy.

Anyone care to suggest an approach to making this faster?

-M

Re: Speeding up highlighting by storing a cached TokenStream

[Mark Miller <ma...@gmail.com>]

Test the difference though. With small docs I experience better speed 
just re-analyzing...if the docs are big enough TokenSources starts to 
show an advantage...gotto be pretty big though. Your millage may vary 
though, so best to check.

markharw00d wrote:
>
>> Anyone care to suggest an approach to making this faster?
>>   
> See TokenSources.java
>
>
> Cheers
> Mark
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Speeding up highlighting by storing a cached TokenStream

[markharw00d <ma...@yahoo.co.uk>]

> Anyone care to suggest an approach to making this faster?
>   
See TokenSources.java


Cheers
 Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org