You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Victor Solakhian <vs...@gmail.com> on 2017/04/25 15:34:17 UTC
SolrIndexSearcher#getDocList() method returns zero results, if query
includes tdate range query
We have code that uses *SolrIndexSearcher#getDocList()* method to get
document IDs for the query.
First a Solr query string is generated from UI, then the following code
creates a Lucene Query
org.apache.lucene.search.Query query = parser.parse(solrQueryString);
where parser is org.apache.lucene.queryparser.classic.QueryParser and then
the following is used to get the document IDs:
DocList docList = indexSearcher.getDocList(query, filterList, sort,
start, length, 0);
The code worked perfectly in Solr 4.5. Now, in Solr 5.5.4, it works only if
the query does not contain a date range query. For example, solrQueryString:
"(+c_class:(Industry.government)) AND +valid_date:[2015-10-21 TO
2017-04-21] AND -class:(TitleCodeMiddle.Board) AND +company:[* TO *] AND
+(has_email:(true) OR has_phone:(true) OR c_has_phone:(true)) AND
+c_class:(Industry.government)"
was parsed to the Lucene query:
"+(+c_class:industry.government) +valid_date:[2015-10-21 TO 2017-04-21]
-class:titlecodemiddle.board +company:[* TO *] +(has_email:T has_phone:T
c_has_phone:T) +c_class:industry.government"
which contains "+valid_date:[2015-10-21 TO 2017-04-21]", returns ZERO
results, although the same query (actually the Solr equivalent) returns
3326 records when used in Solr Admin.
Here is the definition of the "valid_date" field:
<field name="valid_date" type="tdate" indexed="true" stored="false" />
<fieldType name="tdate" class="solr.TrieDateField"
sortMissingLast="true" precisionStep="6" positionIncrementGap="0"
omitNorms="true"/>
For a similar query without the range query:
+(+c_class:industry.government) -class:titlecodemiddle.board +company:[* TO
*] +(has_email:T has_phone:T c_has_phone:T) +c_class:industry.government
our code returns 5629 results (same as Solr Admin).
I tried to use different formats for date in the Solr query (according to
what I was able to find on the web for Lucene date format):
- "+valid_date:[2015-10-21 TO 2017-04-21]"
- "+valid_date:[20151021 TO 20170421]"
- "+valid_date:[2015-10-21T04:00:00.000 TO 2017-04-21T04:00:00.000]"
- "+valid_date:[2015-10-21T04:00:00.000Z TO 2017-04-21T04\:00\:00]"
- "+valid_date:[2015-10-21T00:00:00Z TO 2017-04-21T00:00:00Z]"
Just out of curiosity, I even generated "+valid_date:[XXX TO XXX]" just to
see that SolrIndexSearcher#getDocList() method does not check for correct
syntax and returns ZERO results.
Does anybody know what is happening and what is the proper date format for
Lucene range query in v. 5.5.4?
Thanks,
Victor
Re: SolrIndexSearcher#getDocList() method returns zero results, if
query includes tdate range query
Posted by Victor Solakhian <vs...@gmail.com>.
I used org.apache.solr.search.LuceneQParser instead of
org.apache.lucene.queryparser.classic.QueryParser and now our code works.
Here are some excerpts:
...
QParser qParser = getParser(core, solrQueryString);
Query query = qParser.parse();
...
private QParser getParser(final SolrCore corer, final String queryStr) {
SolrQueryRequest req = new LocalSolrQueryRequest(core, queryStr,
null, 0, -1, new HashMap());
QParser qParser = null;
try {
qParser = LuceneQParser.getParser(queryStr, null, req);
} catch (SyntaxError e) {
log.error("Failed to get LuceneQParser", e);
}
return qParser;
}
Thanks again,
Victor
On Tue, Apr 25, 2017 at 6:19 PM, Victor Solakhian <vs...@gmail.com>
wrote:
> Hi Chris,
>
> I think you are leading me to the right direction.
>
> I'm not sure how that would have worked in Solr 4.5, ... unless
>> perhaps your definition of a "date" field was different in the schema's
>> you used in 4.5, and did not involve a Trie based date field (the very
>> old legacy date format ields used a simple String based encoding that
>> might have worked)
>
>
> You are right. In Solr 4.5 we had:
>
> <field name="valid_date" type="pdate" indexed="true" stored="false" />
> <fieldType name="pdate" class="solr.DateField" sortMissingLast="true"
> omitNorms="true"/>
>
> I will need some time to digest all information you provided. I will let
> you know.
>
> Thank you very much.
>
> Victor
>
>
>
> On Tue, Apr 25, 2017 at 2:45 PM, Chris Hostetter <hossman_lucene@fucit.org
> > wrote:
>
>> : The correct way for a plugin to do the sort of thing you are trying to
>> do
>> : would be to use an instance of SolrQueryParser -- see for example the
>> code
>> : in LuceneQParser and how it uses SolrQueryParser ... you'll most likeley
>> : just want to use LuceneQParser directly in your plugin to simplify
>> things.
>>
>> ...or depending on how low level you want to deal with things, consider
>> using IndexSchema.getField(...).getFieldType().getRangeQuery(null, ...)
>> to
>> build your range Query object directly from the low/high end points
>> provided as input instead of needing to build a string just to parse it
>> again.
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>
>
Re: SolrIndexSearcher#getDocList() method returns zero results, if
query includes tdate range query
Posted by Victor Solakhian <vs...@gmail.com>.
Hi Chris,
I think you are leading me to the right direction.
I'm not sure how that would have worked in Solr 4.5, ... unless
> perhaps your definition of a "date" field was different in the schema's
> you used in 4.5, and did not involve a Trie based date field (the very
> old legacy date format ields used a simple String based encoding that
> might have worked)
You are right. In Solr 4.5 we had:
<field name="valid_date" type="pdate" indexed="true" stored="false" />
<fieldType name="pdate" class="solr.DateField" sortMissingLast="true"
omitNorms="true"/>
I will need some time to digest all information you provided. I will let
you know.
Thank you very much.
Victor
On Tue, Apr 25, 2017 at 2:45 PM, Chris Hostetter <ho...@fucit.org>
wrote:
> : The correct way for a plugin to do the sort of thing you are trying to do
> : would be to use an instance of SolrQueryParser -- see for example the
> code
> : in LuceneQParser and how it uses SolrQueryParser ... you'll most likeley
> : just want to use LuceneQParser directly in your plugin to simplify
> things.
>
> ...or depending on how low level you want to deal with things, consider
> using IndexSchema.getField(...).getFieldType().getRangeQuery(null, ...) to
> build your range Query object directly from the low/high end points
> provided as input instead of needing to build a string just to parse it
> again.
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: SolrIndexSearcher#getDocList() method returns zero results, if
query includes tdate range query
Posted by Chris Hostetter <ho...@fucit.org>.
: The correct way for a plugin to do the sort of thing you are trying to do
: would be to use an instance of SolrQueryParser -- see for example the code
: in LuceneQParser and how it uses SolrQueryParser ... you'll most likeley
: just want to use LuceneQParser directly in your plugin to simplify things.
...or depending on how low level you want to deal with things, consider
using IndexSchema.getField(...).getFieldType().getRangeQuery(null, ...) to
build your range Query object directly from the low/high end points
provided as input instead of needing to build a string just to parse it
again.
-Hoss
http://www.lucidworks.com/
Re: SolrIndexSearcher#getDocList() method returns zero results, if
query includes tdate range query
Posted by Chris Hostetter <ho...@fucit.org>.
Diff FieldType's encode diff values into terms in diff ways. at query
time the FieldTypes need to be consulted to know how to build the
resulting query object.
Solr's query parsers are "schema aware" and delegate to the appropriate
FieldType to handle any index term encoding needed -- but the lower level
lucene QueryParser that you ar using does not...
: org.apache.lucene.search.Query query = parser.parse(solrQueryString);
:
: where parser is org.apache.lucene.queryparser.classic.QueryParser and then
: the following is used to get the document IDs:
...for anything except trivial StrField instances, that's not going to
work -- not for any Trie based fields, or any TextFields (unless you go
get the Analyzer from the schema) or any non trivial FieldTypes.
: The code worked perfectly in Solr 4.5. Now, in Solr 5.5.4, it works only if
: the query does not contain a date range query. For example, solrQueryString:
I'm not sure how that would have worked in Solr 4.5, ... unless
perhaps your definition of a "date" field was different in the schema's
you used in 4.5, and did not involve a Trie based date field (the very
old legacy date format ields used a simple String based encoding that
might have worked)
The correct way for a plugin to do the sort of thing you are trying to do
would be to use an instance of SolrQueryParser -- see for example the code
in LuceneQParser and how it uses SolrQueryParser ... you'll most likeley
just want to use LuceneQParser directly in your plugin to simplify things.
-Hoss
http://www.lucidworks.com/
Re: SolrIndexSearcher#getDocList() method returns zero results, if query includes tdate range query
Posted by Rick Leir <rl...@leirtech.com>.
Victor,
In SorAdmin, do a query then look at the top bar on the screen. Sorry, I cannot do a screenshot here. The actual query that SolrAdmin generated is in that top bar. It is difficult to cut and paste the query but possible. Or you can click on it and jump to a results page. SolrAdmin has changed over time, I know the 5.5.2 version best.
GetDocList must cause a query to get sent? You could see the query in the logs or even use Wireshark.
Cheers -- Rick
On April 25, 2017 1:11:25 PM EDT, Victor Solakhian <vs...@gmail.com> wrote:
>Rick,
>
>Solr Admin does not generate a query. I use it just to confirm that the
>query generated by our code returns results.
>
>getDocList() method also does not generate a query, It returns a list
>of
>document IDs for the query created by the QueryParser.parse(query,...).
>method.
>
>Thanks,
>
>Victor
>
>On Tue, Apr 25, 2017 at 12:44 PM, Rick Leir <rl...@leirtech.com> wrote:
>
>> Victor,
>> When you do a query in SolrAdmin, the generated query is shown in at
>the
>> top of the page. Can you compare that with the query that getDocList
>> generates? Or did I misunderstand your question.
>> Cheers -- Rick
>>
>> On April 25, 2017 11:34:17 AM EDT, Victor Solakhian
><vs...@gmail.com>
>> wrote:
>> >We have code that uses *SolrIndexSearcher#getDocList()* method to
>get
>> >document IDs for the query.
>> >
>> >First a Solr query string is generated from UI, then the following
>code
>> >creates a Lucene Query
>> >
>> > org.apache.lucene.search.Query query =
>parser.parse(solrQueryString);
>> >
>> >where parser is org.apache.lucene.queryparser.classic.QueryParser
>and
>> >then
>> >the following is used to get the document IDs:
>> >
>> > DocList docList = indexSearcher.getDocList(query, filterList,
>sort,
>> >start, length, 0);
>> >
>> >The code worked perfectly in Solr 4.5. Now, in Solr 5.5.4, it works
>> >only if
>> >the query does not contain a date range query. For example,
>> >solrQueryString:
>> >
>> >"(+c_class:(Industry.government)) AND +valid_date:[2015-10-21 TO
>> >2017-04-21] AND -class:(TitleCodeMiddle.Board) AND +company:[* TO *]
>> >AND
>> >+(has_email:(true) OR has_phone:(true) OR c_has_phone:(true)) AND
>> >+c_class:(Industry.government)"
>> >
>> >was parsed to the Lucene query:
>> >
>> >"+(+c_class:industry.government) +valid_date:[2015-10-21 TO
>2017-04-21]
>> >-class:titlecodemiddle.board +company:[* TO *] +(has_email:T
>> >has_phone:T
>> >c_has_phone:T) +c_class:industry.government"
>> >
>> >which contains "+valid_date:[2015-10-21 TO 2017-04-21]", returns
>ZERO
>> >results, although the same query (actually the Solr equivalent)
>returns
>> >3326 records when used in Solr Admin.
>> >
>> >Here is the definition of the "valid_date" field:
>> >
>> > <field name="valid_date" type="tdate" indexed="true" stored="false"
>/>
>> >
>> > <fieldType name="tdate" class="solr.TrieDateField"
>> >sortMissingLast="true" precisionStep="6" positionIncrementGap="0"
>> >omitNorms="true"/>
>> >
>> >For a similar query without the range query:
>> >
>> >+(+c_class:industry.government) -class:titlecodemiddle.board
>> >+company:[* TO
>> >*] +(has_email:T has_phone:T c_has_phone:T)
>> >+c_class:industry.government
>> >
>> >our code returns 5629 results (same as Solr Admin).
>> >
>> >I tried to use different formats for date in the Solr query
>(according
>> >to
>> >what I was able to find on the web for Lucene date format):
>> >
>> > - "+valid_date:[2015-10-21 TO 2017-04-21]"
>> > - "+valid_date:[20151021 TO 20170421]"
>> > - "+valid_date:[2015-10-21T04:00:00.000 TO
>2017-04-21T04:00:00.000]"
>> > - "+valid_date:[2015-10-21T04:00:00.000Z TO
>2017-04-21T04\:00\:00]"
>> > - "+valid_date:[2015-10-21T00:00:00Z TO 2017-04-21T00:00:00Z]"
>> >
>> >Just out of curiosity, I even generated "+valid_date:[XXX TO XXX]"
>just
>> >to
>> >see that SolrIndexSearcher#getDocList() method does not check for
>> >correct
>> >syntax and returns ZERO results.
>> >
>> >Does anybody know what is happening and what is the proper date
>format
>> >for
>> >Lucene range query in v. 5.5.4?
>> >
>> >Thanks,
>> >
>> >Victor
>>
>> --
>> Sorry for being brief. Alternate email is rickleir at yahoo dot com
--
Sorry for being brief. Alternate email is rickleir at yahoo dot com
Re: SolrIndexSearcher#getDocList() method returns zero results, if
query includes tdate range query
Posted by Victor Solakhian <vs...@gmail.com>.
Rick,
Solr Admin does not generate a query. I use it just to confirm that the
query generated by our code returns results.
getDocList() method also does not generate a query, It returns a list of
document IDs for the query created by the QueryParser.parse(query,...).
method.
Thanks,
Victor
On Tue, Apr 25, 2017 at 12:44 PM, Rick Leir <rl...@leirtech.com> wrote:
> Victor,
> When you do a query in SolrAdmin, the generated query is shown in at the
> top of the page. Can you compare that with the query that getDocList
> generates? Or did I misunderstand your question.
> Cheers -- Rick
>
> On April 25, 2017 11:34:17 AM EDT, Victor Solakhian <vs...@gmail.com>
> wrote:
> >We have code that uses *SolrIndexSearcher#getDocList()* method to get
> >document IDs for the query.
> >
> >First a Solr query string is generated from UI, then the following code
> >creates a Lucene Query
> >
> > org.apache.lucene.search.Query query = parser.parse(solrQueryString);
> >
> >where parser is org.apache.lucene.queryparser.classic.QueryParser and
> >then
> >the following is used to get the document IDs:
> >
> > DocList docList = indexSearcher.getDocList(query, filterList, sort,
> >start, length, 0);
> >
> >The code worked perfectly in Solr 4.5. Now, in Solr 5.5.4, it works
> >only if
> >the query does not contain a date range query. For example,
> >solrQueryString:
> >
> >"(+c_class:(Industry.government)) AND +valid_date:[2015-10-21 TO
> >2017-04-21] AND -class:(TitleCodeMiddle.Board) AND +company:[* TO *]
> >AND
> >+(has_email:(true) OR has_phone:(true) OR c_has_phone:(true)) AND
> >+c_class:(Industry.government)"
> >
> >was parsed to the Lucene query:
> >
> >"+(+c_class:industry.government) +valid_date:[2015-10-21 TO 2017-04-21]
> >-class:titlecodemiddle.board +company:[* TO *] +(has_email:T
> >has_phone:T
> >c_has_phone:T) +c_class:industry.government"
> >
> >which contains "+valid_date:[2015-10-21 TO 2017-04-21]", returns ZERO
> >results, although the same query (actually the Solr equivalent) returns
> >3326 records when used in Solr Admin.
> >
> >Here is the definition of the "valid_date" field:
> >
> > <field name="valid_date" type="tdate" indexed="true" stored="false" />
> >
> > <fieldType name="tdate" class="solr.TrieDateField"
> >sortMissingLast="true" precisionStep="6" positionIncrementGap="0"
> >omitNorms="true"/>
> >
> >For a similar query without the range query:
> >
> >+(+c_class:industry.government) -class:titlecodemiddle.board
> >+company:[* TO
> >*] +(has_email:T has_phone:T c_has_phone:T)
> >+c_class:industry.government
> >
> >our code returns 5629 results (same as Solr Admin).
> >
> >I tried to use different formats for date in the Solr query (according
> >to
> >what I was able to find on the web for Lucene date format):
> >
> > - "+valid_date:[2015-10-21 TO 2017-04-21]"
> > - "+valid_date:[20151021 TO 20170421]"
> > - "+valid_date:[2015-10-21T04:00:00.000 TO 2017-04-21T04:00:00.000]"
> > - "+valid_date:[2015-10-21T04:00:00.000Z TO 2017-04-21T04\:00\:00]"
> > - "+valid_date:[2015-10-21T00:00:00Z TO 2017-04-21T00:00:00Z]"
> >
> >Just out of curiosity, I even generated "+valid_date:[XXX TO XXX]" just
> >to
> >see that SolrIndexSearcher#getDocList() method does not check for
> >correct
> >syntax and returns ZERO results.
> >
> >Does anybody know what is happening and what is the proper date format
> >for
> >Lucene range query in v. 5.5.4?
> >
> >Thanks,
> >
> >Victor
>
> --
> Sorry for being brief. Alternate email is rickleir at yahoo dot com
Re: SolrIndexSearcher#getDocList() method returns zero results, if query includes tdate range query
Posted by Rick Leir <rl...@leirtech.com>.
Victor,
When you do a query in SolrAdmin, the generated query is shown in at the top of the page. Can you compare that with the query that getDocList generates? Or did I misunderstand your question.
Cheers -- Rick
On April 25, 2017 11:34:17 AM EDT, Victor Solakhian <vs...@gmail.com> wrote:
>We have code that uses *SolrIndexSearcher#getDocList()* method to get
>document IDs for the query.
>
>First a Solr query string is generated from UI, then the following code
>creates a Lucene Query
>
> org.apache.lucene.search.Query query = parser.parse(solrQueryString);
>
>where parser is org.apache.lucene.queryparser.classic.QueryParser and
>then
>the following is used to get the document IDs:
>
> DocList docList = indexSearcher.getDocList(query, filterList, sort,
>start, length, 0);
>
>The code worked perfectly in Solr 4.5. Now, in Solr 5.5.4, it works
>only if
>the query does not contain a date range query. For example,
>solrQueryString:
>
>"(+c_class:(Industry.government)) AND +valid_date:[2015-10-21 TO
>2017-04-21] AND -class:(TitleCodeMiddle.Board) AND +company:[* TO *]
>AND
>+(has_email:(true) OR has_phone:(true) OR c_has_phone:(true)) AND
>+c_class:(Industry.government)"
>
>was parsed to the Lucene query:
>
>"+(+c_class:industry.government) +valid_date:[2015-10-21 TO 2017-04-21]
>-class:titlecodemiddle.board +company:[* TO *] +(has_email:T
>has_phone:T
>c_has_phone:T) +c_class:industry.government"
>
>which contains "+valid_date:[2015-10-21 TO 2017-04-21]", returns ZERO
>results, although the same query (actually the Solr equivalent) returns
>3326 records when used in Solr Admin.
>
>Here is the definition of the "valid_date" field:
>
> <field name="valid_date" type="tdate" indexed="true" stored="false" />
>
> <fieldType name="tdate" class="solr.TrieDateField"
>sortMissingLast="true" precisionStep="6" positionIncrementGap="0"
>omitNorms="true"/>
>
>For a similar query without the range query:
>
>+(+c_class:industry.government) -class:titlecodemiddle.board
>+company:[* TO
>*] +(has_email:T has_phone:T c_has_phone:T)
>+c_class:industry.government
>
>our code returns 5629 results (same as Solr Admin).
>
>I tried to use different formats for date in the Solr query (according
>to
>what I was able to find on the web for Lucene date format):
>
> - "+valid_date:[2015-10-21 TO 2017-04-21]"
> - "+valid_date:[20151021 TO 20170421]"
> - "+valid_date:[2015-10-21T04:00:00.000 TO 2017-04-21T04:00:00.000]"
> - "+valid_date:[2015-10-21T04:00:00.000Z TO 2017-04-21T04\:00\:00]"
> - "+valid_date:[2015-10-21T00:00:00Z TO 2017-04-21T00:00:00Z]"
>
>Just out of curiosity, I even generated "+valid_date:[XXX TO XXX]" just
>to
>see that SolrIndexSearcher#getDocList() method does not check for
>correct
>syntax and returns ZERO results.
>
>Does anybody know what is happening and what is the proper date format
>for
>Lucene range query in v. 5.5.4?
>
>Thanks,
>
>Victor
--
Sorry for being brief. Alternate email is rickleir at yahoo dot com