You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jon Pipitone <jp...@mshri.on.ca> on 2003/05/12 17:23:02 UTC
'-' character not interpreted correctly in field names
Hi all,
> I believe that the tokenizer treats a dash as a token separator.
> Hence, the only way, as I recall, to eliminate this behavior is
> to modify QueryParser.jj so it doesn't do this. However, doing
> this can cause some other problems, like hyphenated words at a
> line break and the like.
I've recently started using lucene and I'm running into the same issue
with the query parser. I'd like to use queries that contain dashes in
the field name, but as far as I can tell it seems that the current query
grammar treats field names as terms, and so, as Terry notes, a dash
becomes a token seperator.
Terry suggests modifying the QueryParser.jj -- I would suspect by
creating a seperate non-terminal for field names.
Has anyone done any work on this already? Is modifying QueryParser.jj
the best approach?
Thanks,
jp
> From: Terry Steichen <te...@net-frame.com>
> Subject: '-' character not interpreted correctly in field names
> Date: Mon, 3 Feb 2003 09:19:58 -0500
> Content-Type: text/plain;
> charset="iso-8859-1"
>
>
> I believe that the tokenizer treats a dash as a token separator. Hence, the
> only way, as I recall, to eliminate this behavior is to modify
> QueryParser.jj so it doesn't do this. However, doing this can cause some
> other problems, like hyphenated words at a line break and the like.
>
> (Of course, if you do make such a change, you'll have to go back and reindex
> after such a change.)
>
> I've run into this problem myself and I've 'punted' - on certain fields,
> when I index, I replace the dash with an underscore. This isn't a real good
> solution, and it does require me to keep remembering in which fields I have
> to do this substitution in the search. But, for the moment it works. I'll
> probably go back and make some kind of change later, when I have more time.
>
> HTH,
>
> Terry
>
> ----- Original Message -----
> From: "hermit" <he...@freestart.hu>
> To: <lu...@jakarta.apache.org>
> Sent: Monday, February 03, 2003 2:39 AM
> Subject: '-' character not interpreted correctly in field names
>
>
>> Hello!
>>
>> I have a problem, a big one. I have successfully indexed 600 MB of XML
>> data, but the search can't give any results if the field contains any
>> '-' characters .
>> For example: compound@cgx-code:[2 - 5] must match at least two results
>> based on my XML data but it gives nothing.
>>
>> Can you advice me a simple solution? Or is it a bug?
>>
>> The Hermit
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: '-' character not interpreted correctly in field names
Posted by Jon Pipitone <jp...@mshri.on.ca>.
Hi Otis, Terry,
>>>You can write a custom Analyzer that does not remove dashes from
>>>tokens, and use it for both indexing and searching.
>>>
>>>This is a frequent question and answer on this list.
Sorry for the noise, but I haven't been able to find a solution in the
mailing list archives, or by writing my own analyzer:
public class MyAnalyzer extends Analyzer {
public TokenStream tokenStream(String fieldName, Reader reader) {
return new CharTokenizer(reader) {
protected boolean isTokenChar(char c) {
return Character.isLetter(c) || c == '-';
}
};
}
}
I parse a query like this:
String queryString = "foo-bar:foo";
String queryResult =
QueryParser.parse(queryString, "body", new MyAnalyzer())
With the output:
body:foo -bar:foo
But I would expect the output:
foo-bar:foo
If I print out the tokens that MyAnalyzer produces I do get "foo-bar"
and then "foo".
Any pointers on what I'm doing wrong?
jp
>>>--- Jon Pipitone <jp...@mshri.on.ca> wrote:
>>>
>>>>Hi all,
>>>>
>>>> > I believe that the tokenizer treats a dash as a token
>>>
>>separator.
>>
>>>> > Hence, the only way, as I recall, to eliminate this behavior
>>>
>>is
>>
>>>> > to modify QueryParser.jj so it doesn't do this. However,
>>>
>>doing
>>
>>>> > this can cause some other problems, like hyphenated words at a
>>>> > line break and the like.
>>>>
>>>>I've recently started using lucene and I'm running into the same
>>>>issue
>>>>with the query parser. I'd like to use queries that contain
>>>
>>dashes
>>
>>>>in
>>>>the field name, but as far as I can tell it seems that the
>>>
>>current
>>
>>>>query
>>>>grammar treats field names as terms, and so, as Terry notes, a
>>>
>>dash
>>
>>>>becomes a token seperator.
>>>>
>>>>Terry suggests modifying the QueryParser.jj -- I would suspect by
>>>>creating a seperate non-terminal for field names.
>>>>
>>>>Has anyone done any work on this already? Is modifying
>>>>QueryParser.jj
>>>>the best approach?
>>>>
>>>>Thanks,
>>>>jp
>>>>
>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: '-' character not interpreted correctly in field names
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Nah, but we could include it in the FAQ or Lucene Sandbox. If you have
something ready to go, please email.
Otis
--- Terry Steichen <te...@net-frame.com> wrote:
> Otis,
>
> Since this does indeed comes up so often, maybe we should include an
> example
> of such an Analyzer with the basic package?
>
> Regards,
>
> Terry
>
> ----- Original Message -----
> From: "Otis Gospodnetic" <ot...@yahoo.com>
> To: "Lucene Users List" <lu...@jakarta.apache.org>
> Sent: Monday, May 12, 2003 11:41 AM
> Subject: Re: '-' character not interpreted correctly in field names
>
>
> > You can write a custom Analyzer that does not remove dashes from
> > tokens, and use it for both indexing and searching.
> >
> > This is a frequent question and answer on this list.
> >
> > Otis
> >
> > --- Jon Pipitone <jp...@mshri.on.ca> wrote:
> > > Hi all,
> > >
> > > > I believe that the tokenizer treats a dash as a token
> separator.
> > > > Hence, the only way, as I recall, to eliminate this behavior
> is
> > > > to modify QueryParser.jj so it doesn't do this. However,
> doing
> > > > this can cause some other problems, like hyphenated words at a
> > > > line break and the like.
> > >
> > > I've recently started using lucene and I'm running into the same
> > > issue
> > > with the query parser. I'd like to use queries that contain
> dashes
> > > in
> > > the field name, but as far as I can tell it seems that the
> current
> > > query
> > > grammar treats field names as terms, and so, as Terry notes, a
> dash
> > > becomes a token seperator.
> > >
> > > Terry suggests modifying the QueryParser.jj -- I would suspect by
> > > creating a seperate non-terminal for field names.
> > >
> > > Has anyone done any work on this already? Is modifying
> > > QueryParser.jj
> > > the best approach?
> > >
> > > Thanks,
> > > jp
> > >
> > > > From: Terry Steichen <te...@net-frame.com>
> > > > Subject: '-' character not interpreted correctly in field names
> > > > Date: Mon, 3 Feb 2003 09:19:58 -0500
> > > > Content-Type: text/plain;
> > > > charset="iso-8859-1"
> > > >
> > > >
> > > > I believe that the tokenizer treats a dash as a token
> separator.
> > > Hence, the
> > > > only way, as I recall, to eliminate this behavior is to modify
> > > > QueryParser.jj so it doesn't do this. However, doing this can
> > > cause some
> > > > other problems, like hyphenated words at a line break and the
> like.
> > > >
> > > > (Of course, if you do make such a change, you'll have to go
> back
> > > and reindex
> > > > after such a change.)
> > > >
> > > > I've run into this problem myself and I've 'punted' - on
> certain
> > > fields,
> > > > when I index, I replace the dash with an underscore. This
> isn't a
> > > real good
> > > > solution, and it does require me to keep remembering in which
> > > fields I have
> > > > to do this substitution in the search. But, for the moment it
> > > works. I'll
> > > > probably go back and make some kind of change later, when I
> have
> > > more time.
> > > >
> > > > HTH,
> > > >
> > > > Terry
> > > >
> > > > ----- Original Message -----
> > > > From: "hermit" <he...@freestart.hu>
> > > > To: <lu...@jakarta.apache.org>
> > > > Sent: Monday, February 03, 2003 2:39 AM
> > > > Subject: '-' character not interpreted correctly in field names
> > > >
> > > >
> > > >> Hello!
> > > >>
> > > >> I have a problem, a big one. I have successfully indexed 600
> MB of
> > > XML
> > > >> data, but the search can't give any results if the field
> contains
> > > any
> > > >> '-' characters .
> > > >> For example: compound@cgx-code:[2 - 5] must match at least two
> > > results
> > > >> based on my XML data but it gives nothing.
> > > >>
> > > >> Can you advice me a simple solution? Or is it a bug?
> > > >>
> > > >> The Hermit
> > >
> > >
> > >
> > >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> > >
> >
> >
> > __________________________________
> > Do you Yahoo!?
> > The New Yahoo! Search - Faster. Easier. Bingo.
> > http://search.yahoo.com
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: '-' character not interpreted correctly in field names
Posted by Terry Steichen <te...@net-frame.com>.
Otis,
Since this does indeed comes up so often, maybe we should include an example
of such an Analyzer with the basic package?
Regards,
Terry
----- Original Message -----
From: "Otis Gospodnetic" <ot...@yahoo.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Monday, May 12, 2003 11:41 AM
Subject: Re: '-' character not interpreted correctly in field names
> You can write a custom Analyzer that does not remove dashes from
> tokens, and use it for both indexing and searching.
>
> This is a frequent question and answer on this list.
>
> Otis
>
> --- Jon Pipitone <jp...@mshri.on.ca> wrote:
> > Hi all,
> >
> > > I believe that the tokenizer treats a dash as a token separator.
> > > Hence, the only way, as I recall, to eliminate this behavior is
> > > to modify QueryParser.jj so it doesn't do this. However, doing
> > > this can cause some other problems, like hyphenated words at a
> > > line break and the like.
> >
> > I've recently started using lucene and I'm running into the same
> > issue
> > with the query parser. I'd like to use queries that contain dashes
> > in
> > the field name, but as far as I can tell it seems that the current
> > query
> > grammar treats field names as terms, and so, as Terry notes, a dash
> > becomes a token seperator.
> >
> > Terry suggests modifying the QueryParser.jj -- I would suspect by
> > creating a seperate non-terminal for field names.
> >
> > Has anyone done any work on this already? Is modifying
> > QueryParser.jj
> > the best approach?
> >
> > Thanks,
> > jp
> >
> > > From: Terry Steichen <te...@net-frame.com>
> > > Subject: '-' character not interpreted correctly in field names
> > > Date: Mon, 3 Feb 2003 09:19:58 -0500
> > > Content-Type: text/plain;
> > > charset="iso-8859-1"
> > >
> > >
> > > I believe that the tokenizer treats a dash as a token separator.
> > Hence, the
> > > only way, as I recall, to eliminate this behavior is to modify
> > > QueryParser.jj so it doesn't do this. However, doing this can
> > cause some
> > > other problems, like hyphenated words at a line break and the like.
> > >
> > > (Of course, if you do make such a change, you'll have to go back
> > and reindex
> > > after such a change.)
> > >
> > > I've run into this problem myself and I've 'punted' - on certain
> > fields,
> > > when I index, I replace the dash with an underscore. This isn't a
> > real good
> > > solution, and it does require me to keep remembering in which
> > fields I have
> > > to do this substitution in the search. But, for the moment it
> > works. I'll
> > > probably go back and make some kind of change later, when I have
> > more time.
> > >
> > > HTH,
> > >
> > > Terry
> > >
> > > ----- Original Message -----
> > > From: "hermit" <he...@freestart.hu>
> > > To: <lu...@jakarta.apache.org>
> > > Sent: Monday, February 03, 2003 2:39 AM
> > > Subject: '-' character not interpreted correctly in field names
> > >
> > >
> > >> Hello!
> > >>
> > >> I have a problem, a big one. I have successfully indexed 600 MB of
> > XML
> > >> data, but the search can't give any results if the field contains
> > any
> > >> '-' characters .
> > >> For example: compound@cgx-code:[2 - 5] must match at least two
> > results
> > >> based on my XML data but it gives nothing.
> > >>
> > >> Can you advice me a simple solution? Or is it a bug?
> > >>
> > >> The Hermit
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> >
>
>
> __________________________________
> Do you Yahoo!?
> The New Yahoo! Search - Faster. Easier. Bingo.
> http://search.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: '-' character not interpreted correctly in field names
Posted by Otis Gospodnetic <ot...@yahoo.com>.
You can write a custom Analyzer that does not remove dashes from
tokens, and use it for both indexing and searching.
This is a frequent question and answer on this list.
Otis
--- Jon Pipitone <jp...@mshri.on.ca> wrote:
> Hi all,
>
> > I believe that the tokenizer treats a dash as a token separator.
> > Hence, the only way, as I recall, to eliminate this behavior is
> > to modify QueryParser.jj so it doesn't do this. However, doing
> > this can cause some other problems, like hyphenated words at a
> > line break and the like.
>
> I've recently started using lucene and I'm running into the same
> issue
> with the query parser. I'd like to use queries that contain dashes
> in
> the field name, but as far as I can tell it seems that the current
> query
> grammar treats field names as terms, and so, as Terry notes, a dash
> becomes a token seperator.
>
> Terry suggests modifying the QueryParser.jj -- I would suspect by
> creating a seperate non-terminal for field names.
>
> Has anyone done any work on this already? Is modifying
> QueryParser.jj
> the best approach?
>
> Thanks,
> jp
>
> > From: Terry Steichen <te...@net-frame.com>
> > Subject: '-' character not interpreted correctly in field names
> > Date: Mon, 3 Feb 2003 09:19:58 -0500
> > Content-Type: text/plain;
> > charset="iso-8859-1"
> >
> >
> > I believe that the tokenizer treats a dash as a token separator.
> Hence, the
> > only way, as I recall, to eliminate this behavior is to modify
> > QueryParser.jj so it doesn't do this. However, doing this can
> cause some
> > other problems, like hyphenated words at a line break and the like.
> >
> > (Of course, if you do make such a change, you'll have to go back
> and reindex
> > after such a change.)
> >
> > I've run into this problem myself and I've 'punted' - on certain
> fields,
> > when I index, I replace the dash with an underscore. This isn't a
> real good
> > solution, and it does require me to keep remembering in which
> fields I have
> > to do this substitution in the search. But, for the moment it
> works. I'll
> > probably go back and make some kind of change later, when I have
> more time.
> >
> > HTH,
> >
> > Terry
> >
> > ----- Original Message -----
> > From: "hermit" <he...@freestart.hu>
> > To: <lu...@jakarta.apache.org>
> > Sent: Monday, February 03, 2003 2:39 AM
> > Subject: '-' character not interpreted correctly in field names
> >
> >
> >> Hello!
> >>
> >> I have a problem, a big one. I have successfully indexed 600 MB of
> XML
> >> data, but the search can't give any results if the field contains
> any
> >> '-' characters .
> >> For example: compound@cgx-code:[2 - 5] must match at least two
> results
> >> based on my XML data but it gives nothing.
> >>
> >> Can you advice me a simple solution? Or is it a bug?
> >>
> >> The Hermit
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org