[jira] Commented: (LUCENE-2265) improve automaton performance by running on byte[]

["Robert Muir (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852896#action_12852896 ] 

Robert Muir commented on LUCENE-2265:
-------------------------------------

I discussed this situation with Mike McCandless and I think we might have something of a plan.

For reference, here is the problem: 
* In flex the terms are byte[] (typically UTF-8)
* Automaton transitions work on UTF-16 intervals (char)
* RunAutomaton is an array-compiled form that also works on UTF-16 (char[])
* Because of this, we have a lot of unicode conversion overhead between byte[] and char[] hurting performance.

Here is the current idea:
* Switch Automaton to work on UTF-32 intervals (int)
* Create a method to convert a UTF-32 Automaton to an equivalent UTF-8 Automaton.
* Create a UTF-8 RunAutomaton that works on byte[]
* We could also create a UTF-32 RunAutomaton that works on codepoints, for use in analysis, etc.

This would have some nice benefits besides performance, 
for example a wildcard operator of "?" or regex operator of "." would match a real unicode codepoint, 
not a single code unit like it always has. So if somehow we can make this work, we would have better
performance and better unicode support.

The trick is to do this UTF-32 DFA -> UTF-8 DFA conversion efficiently, especially keeping determinism,
and not causing some nasty explosion


> improve automaton performance by running on byte[]
> --------------------------------------------------
>
>                 Key: LUCENE-2265
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2265
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>    Affects Versions: Flex Branch
>            Reporter: Robert Muir
>            Priority: Minor
>             Fix For: Flex Branch
>
>         Attachments: LUCENE-2265.patch
>
>
> Currently, when enumerating terms, automaton must convert entire terms from flex's native utf-8 byte[] to char[] first, then step each char thru the state machine.
> we can make this more efficient, by allowing the state machine to run on byte[], so it can return true/false faster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org