You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Chongchen Chen (Jira)" <ji...@apache.org> on 2019/10/10 03:21:00 UTC
[jira] [Comment Edited] (LUCENE-8137) GraphTokenStreamFiniteStrings
does not handle position inc > 1 in multi-word synoyms
[ https://issues.apache.org/jira/browse/LUCENE-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948157#comment-16948157 ]
Chongchen Chen edited comment on LUCENE-8137 at 10/10/19 3:20 AM:
------------------------------------------------------------------
In QueryBuilder.analyzeGraphBoolean, it create GraphTokenStreamFiniteStrings from input token stream first. GraphTokenStreamFiniteStrings will build an automata from the graph, it will trim the graph. in test case the graph before trim is
```
guinea
0 -------> 1 2
|----------------|
cavy
# 2 is the finish state.
```
but state 1 cannot reach finish state. so it's trimmed. code is in GraphTokenStreamFiniteStrings.java
```java
public GraphTokenStreamFiniteStrings(TokenStream in) throws IOException {
Automaton aut = build(in);
this.det = Operations.removeDeadStates(Operations.determinize(aut, DEFAULT_MAX_DETERMINIZED_STATES));
}
```
was (Author: nppoly):
In QueryBuilder.analyzeGraphBoolean, it create GraphTokenStreamFiniteStrings from input token stream first. GraphTokenStreamFiniteStrings will build an automata from the graph, it will trim the graph. in test case the graph before trim is
```
guinea
0 -------> 1 2
|___________________|
cavy
# 2 is the finish state.
```
but state 1 cannot reach finish state. so it's trimmed. code is in GraphTokenStreamFiniteStrings.java
```java
public GraphTokenStreamFiniteStrings(TokenStream in) throws IOException {
Automaton aut = build(in);
this.det = Operations.removeDeadStates(Operations.determinize(aut, DEFAULT_MAX_DETERMINIZED_STATES));
}
```
> GraphTokenStreamFiniteStrings does not handle position inc > 1 in multi-word synoyms
> ------------------------------------------------------------------------------------
>
> Key: LUCENE-8137
> URL: https://issues.apache.org/jira/browse/LUCENE-8137
> Project: Lucene - Core
> Issue Type: Bug
> Affects Versions: 7.2.1, 8.0
> Reporter: Jim Ferenczi
> Assignee: Jim Ferenczi
> Priority: Major
> Attachments: SGF_SF_interaction.patch
>
>
> The automaton built for graph queries that contain multiple multi-word synonyms does not handle gaps if they appear in the middle of a multi-word synonym. In such case the token next to the gap is considered as part of the multi-word synonym.
> Stop words that appear before or after multi-word synonyms are handled correctly in the current version but the synonym rule "part of speech, pos" for instance does not create the expected query if "of" is removed by a filter that is set after the synonym_graph. One solution would be to reuse TokenStreamToAutomaton (with minor changes to add the ability to create token transitions rather than chars) which preserves gaps (as a transition) in the produced automaton.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org