You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Markus Jelsma <ma...@openindex.io> on 2012/11/08 14:01:37 UTC

positions and qf parameter in (e)dismax

Hi,

We do not want to store positions for some fields or omit term and positions (or just tf) for other fields. Obviously we don't need/want explicit phrase matching on the fields we want to configure without positions, but (e)dismax doesn't let us. All text fields configured in the QF parameter are eligible for explicit phrase matching and need to have positions. We're looking for a way to disable what we don't need and prevent Solr from searching fields for phrases that we don't want to be searched on.

Essentially we'd want to limit explicit phrase matching to the same fields configured in pf or have Lucene ignore explicit phrase searching on fields that have no positions loaded.

Any ideas to share?

Thanks,
Markus

Re: positions and qf parameter in (e)dismax

Posted by Jack Krupansky <ja...@basetechnology.com>.
Edismax is considered Solr, although the same issue exists in the Lucene 
query parser.

-- Jack Krupansky

-----Original Message----- 
From: Markus Jelsma
Sent: Wednesday, November 28, 2012 8:50 AM
To: solr-user@lucene.apache.org
Subject: RE: positions and qf parameter in (e)dismax

I think i agree. Is this something that should be resolved in Solr or 
Lucene?

Thanks

-----Original message-----
> From:Jack Krupansky <ja...@basetechnology.com>
> Sent: Tue 27-Nov-2012 17:47
> To: solr-user@lucene.apache.org
> Subject: Re: positions and qf parameter in (e)dismax
>
> That is exactly the exception I would expect to see given your scenario. 
> The
> fact that the situation may have been silently ignored in the past was
> surely an oversight that has since been corrected.
>
> My proposal would be that if the query parser (actually the code that
> generates PhraseQuery) encounters a phrase and the field does not have
> positions, a BooleanQuery with MUST would be generated instead of the
> PhraseQuery.
>
> -- Jack Krupansky
>
> -----Original Message----- 
> From: Markus Jelsma
> Sent: Tuesday, November 27, 2012 4:27 AM
> To: solr-user@lucene.apache.org
> Subject: RE: positions and qf parameter in (e)dismax
>
> Hi - no we're not getting any errors because we enabled positions on all
> fields that are also listed in the qf-parameter. If we don't, and send a
> phrase query we would get an error such as:
>
> java.lang.IllegalStateException: field "h1" was indexed without position
> data; cannot run
> PhraseQuery (term=a)
> at
> org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:274)
> at
> org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:589)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:280)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1518)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1265)
> at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:384)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:411)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:204)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1555)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:442)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:263)
>
> We have fields in qf that we don't need to be searched for explicit phrase
> queries so we'd like to omit positions, offsets and so on. But if we do,
> we'll get the above error for explicit phrase queries.
>
> Thanks
>
>
> -----Original message-----
> > From:Chris Hostetter <ho...@fucit.org>
> > Sent: Tue 27-Nov-2012 02:15
> > To: solr-user@lucene.apache.org
> > Subject: Re: positions and qf parameter in (e)dismax
> >
> >
> > : We do not want to store positions for some fields or omit term and
> > : positions (or just tf) for other fields. Obviously we don't need/want
> > : explicit phrase matching on the fields we want to configure without
> > : positions, but (e)dismax doesn't let us. All text fields configured in
> > : the QF parameter are eligible for explicit phrase matching and need to
> > : have positions. We're looking for a way to disable what we don't need
> > : and prevent Solr from searching fields for phrases that we don't want 
> > to
> > : be searched on.
> >
> > I'm not understanding the problem ... is there a specific error you are
> > getting? can you please post that error along with your schema and an
> > example of a request that triggers the problem?
> >
> > -Hoss
> >
>
>

RE: positions and qf parameter in (e)dismax

Posted by Markus Jelsma <ma...@openindex.io>.
I think i agree. Is this something that should be resolved in Solr or Lucene?

Thanks 
 
-----Original message-----
> From:Jack Krupansky <ja...@basetechnology.com>
> Sent: Tue 27-Nov-2012 17:47
> To: solr-user@lucene.apache.org
> Subject: Re: positions and qf parameter in (e)dismax
> 
> That is exactly the exception I would expect to see given your scenario. The 
> fact that the situation may have been silently ignored in the past was 
> surely an oversight that has since been corrected.
> 
> My proposal would be that if the query parser (actually the code that 
> generates PhraseQuery) encounters a phrase and the field does not have 
> positions, a BooleanQuery with MUST would be generated instead of the 
> PhraseQuery.
> 
> -- Jack Krupansky
> 
> -----Original Message----- 
> From: Markus Jelsma
> Sent: Tuesday, November 27, 2012 4:27 AM
> To: solr-user@lucene.apache.org
> Subject: RE: positions and qf parameter in (e)dismax
> 
> Hi - no we're not getting any errors because we enabled positions on all 
> fields that are also listed in the qf-parameter. If we don't, and send a 
> phrase query we would get an error such as:
> 
> java.lang.IllegalStateException: field "h1" was indexed without position 
> data; cannot run
> PhraseQuery (term=a)
> at 
> org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:274)
> at 
> org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:589)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:280)
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1518)
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1265)
> at 
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:384)
> at 
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:411)
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:204)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1555)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:442)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:263)
> 
> We have fields in qf that we don't need to be searched for explicit phrase 
> queries so we'd like to omit positions, offsets and so on. But if we do, 
> we'll get the above error for explicit phrase queries.
> 
> Thanks
> 
> 
> -----Original message-----
> > From:Chris Hostetter <ho...@fucit.org>
> > Sent: Tue 27-Nov-2012 02:15
> > To: solr-user@lucene.apache.org
> > Subject: Re: positions and qf parameter in (e)dismax
> >
> >
> > : We do not want to store positions for some fields or omit term and
> > : positions (or just tf) for other fields. Obviously we don't need/want
> > : explicit phrase matching on the fields we want to configure without
> > : positions, but (e)dismax doesn't let us. All text fields configured in
> > : the QF parameter are eligible for explicit phrase matching and need to
> > : have positions. We're looking for a way to disable what we don't need
> > : and prevent Solr from searching fields for phrases that we don't want to
> > : be searched on.
> >
> > I'm not understanding the problem ... is there a specific error you are
> > getting? can you please post that error along with your schema and an
> > example of a request that triggers the problem?
> >
> > -Hoss
> > 
> 
>

Re: positions and qf parameter in (e)dismax

Posted by Jack Krupansky <ja...@basetechnology.com>.
That is exactly the exception I would expect to see given your scenario. The 
fact that the situation may have been silently ignored in the past was 
surely an oversight that has since been corrected.

My proposal would be that if the query parser (actually the code that 
generates PhraseQuery) encounters a phrase and the field does not have 
positions, a BooleanQuery with MUST would be generated instead of the 
PhraseQuery.

-- Jack Krupansky

-----Original Message----- 
From: Markus Jelsma
Sent: Tuesday, November 27, 2012 4:27 AM
To: solr-user@lucene.apache.org
Subject: RE: positions and qf parameter in (e)dismax

Hi - no we're not getting any errors because we enabled positions on all 
fields that are also listed in the qf-parameter. If we don't, and send a 
phrase query we would get an error such as:

java.lang.IllegalStateException: field "h1" was indexed without position 
data; cannot run
PhraseQuery (term=a)
at 
org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:274)
at 
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:589)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:280)
at 
org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1518)
at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1265)
at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:384)
at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:411)
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:204)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1555)
at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:442)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:263)

We have fields in qf that we don't need to be searched for explicit phrase 
queries so we'd like to omit positions, offsets and so on. But if we do, 
we'll get the above error for explicit phrase queries.

Thanks


-----Original message-----
> From:Chris Hostetter <ho...@fucit.org>
> Sent: Tue 27-Nov-2012 02:15
> To: solr-user@lucene.apache.org
> Subject: Re: positions and qf parameter in (e)dismax
>
>
> : We do not want to store positions for some fields or omit term and
> : positions (or just tf) for other fields. Obviously we don't need/want
> : explicit phrase matching on the fields we want to configure without
> : positions, but (e)dismax doesn't let us. All text fields configured in
> : the QF parameter are eligible for explicit phrase matching and need to
> : have positions. We're looking for a way to disable what we don't need
> : and prevent Solr from searching fields for phrases that we don't want to
> : be searched on.
>
> I'm not understanding the problem ... is there a specific error you are
> getting? can you please post that error along with your schema and an
> example of a request that triggers the problem?
>
> -Hoss
>

RE: positions and qf parameter in (e)dismax

Posted by Chris Hostetter <ho...@fucit.org>.
: Hi - no we're not getting any errors because we enabled positions on all 
: fields that are also listed in the qf-parameter. If we don't, and send a 
: phrase query we would get an error such as:
: 
: java.lang.IllegalStateException: field "h1" was indexed without position data; cannot run
: PhraseQuery (term=a)

I'm clearly still missunderstanding something -- probably because i'm 
still not entirely clear on how to reproduce this (you said you are 
omiting positions, but you didn't provide the details on what 
edismax request options you were using were to cause that error).

If you can open a Jira with more details on how to reproduce, we can 
certainly look into it.

: > I'm not understanding the problem ... is there a specific error you are 
: > getting? can you please post that error along with your schema and an 
: > example of a request that triggers the problem?


-Hoss

RE: positions and qf parameter in (e)dismax

Posted by Markus Jelsma <ma...@openindex.io>.
Hi - no we're not getting any errors because we enabled positions on all fields that are also listed in the qf-parameter. If we don't, and send a phrase query we would get an error such as:

java.lang.IllegalStateException: field "h1" was indexed without position data; cannot run
PhraseQuery (term=a)
	at org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:274)
	at org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:589)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:280)
	at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1518)
	at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1265)
	at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:384)
	at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:411)
	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:204)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1555)
	at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:442)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:263)

We have fields in qf that we don't need to be searched for explicit phrase queries so we'd like to omit positions, offsets and so on. But if we do, we'll get the above error for explicit phrase queries.

Thanks
 
 
-----Original message-----
> From:Chris Hostetter <ho...@fucit.org>
> Sent: Tue 27-Nov-2012 02:15
> To: solr-user@lucene.apache.org
> Subject: Re: positions and qf parameter in (e)dismax
> 
> 
> : We do not want to store positions for some fields or omit term and 
> : positions (or just tf) for other fields. Obviously we don't need/want 
> : explicit phrase matching on the fields we want to configure without 
> : positions, but (e)dismax doesn't let us. All text fields configured in 
> : the QF parameter are eligible for explicit phrase matching and need to 
> : have positions. We're looking for a way to disable what we don't need 
> : and prevent Solr from searching fields for phrases that we don't want to 
> : be searched on.
> 
> I'm not understanding the problem ... is there a specific error you are 
> getting? can you please post that error along with your schema and an 
> example of a request that triggers the problem?
> 
> -Hoss
>

Re: positions and qf parameter in (e)dismax

Posted by Chris Hostetter <ho...@fucit.org>.
: We do not want to store positions for some fields or omit term and 
: positions (or just tf) for other fields. Obviously we don't need/want 
: explicit phrase matching on the fields we want to configure without 
: positions, but (e)dismax doesn't let us. All text fields configured in 
: the QF parameter are eligible for explicit phrase matching and need to 
: have positions. We're looking for a way to disable what we don't need 
: and prevent Solr from searching fields for phrases that we don't want to 
: be searched on.

I'm not understanding the problem ... is there a specific error you are 
getting? can you please post that error along with your schema and an 
example of a request that triggers the problem?

-Hoss

RE: positions and qf parameter in (e)dismax

Posted by Markus Jelsma <ma...@openindex.io>.
 
-----Original message-----
> From:Jack Krupansky <ja...@basetechnology.com>
> Sent: Thu 08-Nov-2012 15:56
> To: solr-user@lucene.apache.org
> Subject: Re: positions and qf parameter in (e)dismax
> 
> Sounds like a reasonable request for a new feature to add to Solr.
> 
> Question: Would you want the query to SKIP fields that don't have positions 
> enabled, or to treat a phrase as discrete terms? Or, is that another option 
> you might need to control  for each field?

In the past it was silently ignored. But another parameter or relying on pf seems more appropriate to me.

> 
> -- Jack Krupansky
> 
> -----Original Message----- 
> From: Markus Jelsma
> Sent: Thursday, November 08, 2012 5:01 AM
> To: solr-user@lucene.apache.org
> Subject: positions and qf parameter in (e)dismax
> 
> Hi,
> 
> We do not want to store positions for some fields or omit term and positions 
> (or just tf) for other fields. Obviously we don't need/want explicit phrase 
> matching on the fields we want to configure without positions, but (e)dismax 
> doesn't let us. All text fields configured in the QF parameter are eligible 
> for explicit phrase matching and need to have positions. We're looking for a 
> way to disable what we don't need and prevent Solr from searching fields for 
> phrases that we don't want to be searched on.
> 
> Essentially we'd want to limit explicit phrase matching to the same fields 
> configured in pf or have Lucene ignore explicit phrase searching on fields 
> that have no positions loaded.
> 
> Any ideas to share?
> 
> Thanks,
> Markus 
> 
>

Re: positions and qf parameter in (e)dismax

Posted by Jack Krupansky <ja...@basetechnology.com>.
Sounds like a reasonable request for a new feature to add to Solr.

Question: Would you want the query to SKIP fields that don't have positions 
enabled, or to treat a phrase as discrete terms? Or, is that another option 
you might need to control  for each field?

-- Jack Krupansky

-----Original Message----- 
From: Markus Jelsma
Sent: Thursday, November 08, 2012 5:01 AM
To: solr-user@lucene.apache.org
Subject: positions and qf parameter in (e)dismax

Hi,

We do not want to store positions for some fields or omit term and positions 
(or just tf) for other fields. Obviously we don't need/want explicit phrase 
matching on the fields we want to configure without positions, but (e)dismax 
doesn't let us. All text fields configured in the QF parameter are eligible 
for explicit phrase matching and need to have positions. We're looking for a 
way to disable what we don't need and prevent Solr from searching fields for 
phrases that we don't want to be searched on.

Essentially we'd want to limit explicit phrase matching to the same fields 
configured in pf or have Lucene ignore explicit phrase searching on fields 
that have no positions loaded.

Any ideas to share?

Thanks,
Markus