You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Stanislav Sandalnikov <s....@gmail.com> on 2017/01/24 15:29:09 UTC
Problems with stored/not-stored field filter queries
Hi everyone,
I’m facing strange Solr behavior, which could be better described in examples:
With indexed but not stored IndexDate field:
1) With this query everything works fine, I’m getting the results back:
/select?fl=taskid,docid,score&q=*:*&fq=((((category:"Security")))+AND+(datasource:(sites)))&fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
2) With this query I get nothing:
/select?fl=taskid,docid,score&q=*:*&fq=((((category:"Security")))+AND+(datasource:(sites))+AND+(IndexDate:[NOW/DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
Document’s IndexDate fit specified timeframe for sure.
3) With this query there is no category filter, but there is timeframe filter and everything works fine:
/select?fl=taskid,docid,score&q=*:*&fq=((datasource:(sites))+AND+(IndexDate:[NOW/DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
Then I decided that it might be related to IndexDate as it is not stored. I reindexed data with stored IndexDate field and now query number 2 works just fine.
However, I don’t get the logic here, why it doesn’t work in some particular case? Can someone explain?
Thank you in advance
P.S. category field is indexed but not stored in all cases.
Regards
Stanislav
Re: Problems with stored/not-stored field filter queries
Posted by Erick Erickson <er...@gmail.com>.
bq: By the way is there any way to see if there is a index for some
particular field of a document?
not really conveniently. To know that you have to unwind the inverted
index. The "luke" program can do this. Of course if the field is
_stored_ it's easy, just return q=id:doc_id&fl=*
Also note that in 6x, the above will work for returning the value form
a docValues field even if the field isn't stored,
https://cwiki.apache.org/confluence/display/solr/DocValues. Lots of
background here: https://issues.apache.org/jira/browse/SOLR-8220.
Best,
Erick
On Tue, Jan 24, 2017 at 5:19 PM, Stanislav Sandalnikov
<s....@gmail.com> wrote:
> Thanks Mikhail, didn’t know about debugQuery and explainOther, could be useful.
>
> Regarding $q, you can find this information here - https://cwiki.apache.org/confluence/display/solr/Function+Queries#FunctionQueries-AvailableFunctions <https://cwiki.apache.org/confluence/display/solr/Function+Queries#FunctionQueries-AvailableFunctions> scroll down to «query» function
>
> Stanislav
>
>> 25 янв. 2017 г., в 0:56, Mikhail Khludnev <gg...@gmail.com> написал(а):
>>
>> Hello Stanislav,
>> Stored fields have nothing which findability, I believe. Usually debugQuery
>> and explainOther is a right way to get what's going on there. What is $q ?
>> How it's supposed to work?
>>
>> 24 янв. 2017 г. 18:29 пользователь "Stanislav Sandalnikov" <
>> s.sandalnikov@gmail.com> написал:
>>
>>> Hi everyone,
>>>
>>> I’m facing strange Solr behavior, which could be better described in
>>> examples:
>>>
>>>
>>> With indexed but not stored IndexDate field:
>>>
>>> 1) With this query everything works fine, I’m getting the results back:
>>> /select?fl=taskid,docid,score&q=*:*&fq=((((category:"
>>> Security")))+AND+(datasource:(sites)))&fq={!frange+l%3D0}
>>> query($q)&sort=IndexDate+desc&rows=100&start=0
>>>
>>> 2) With this query I get nothing:
>>> /select?fl=taskid,docid,score&q=*:*&fq=((((category:"
>>> Security")))+AND+(datasource:(sites))+AND+(IndexDate:[NOW/
>>> DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&fq={!frange+l%3D0}query($q)&
>>> sort=IndexDate+desc&rows=100&start=0
>>>
>>> Document’s IndexDate fit specified timeframe for sure.
>>>
>>> 3) With this query there is no category filter, but there is timeframe
>>> filter and everything works fine:
>>> /select?fl=taskid,docid,score&q=*:*&fq=((datasource:(sites))
>>> +AND+(IndexDate:[NOW/DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&
>>> fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
>>>
>>> Then I decided that it might be related to IndexDate as it is not stored.
>>> I reindexed data with stored IndexDate field and now query number 2 works
>>> just fine.
>>>
>>> However, I don’t get the logic here, why it doesn’t work in some
>>> particular case? Can someone explain?
>>>
>>> Thank you in advance
>>>
>>> P.S. category field is indexed but not stored in all cases.
>>>
>>> Regards
>>> Stanislav
>>>
>>>
>>>
>>>
>>>
>
Re: Problems with stored/not-stored field filter queries
Posted by Stanislav Sandalnikov <s....@gmail.com>.
Thanks Mikhail, didn’t know about debugQuery and explainOther, could be useful.
Regarding $q, you can find this information here - https://cwiki.apache.org/confluence/display/solr/Function+Queries#FunctionQueries-AvailableFunctions <https://cwiki.apache.org/confluence/display/solr/Function+Queries#FunctionQueries-AvailableFunctions> scroll down to «query» function
Stanislav
> 25 янв. 2017 г., в 0:56, Mikhail Khludnev <gg...@gmail.com> написал(а):
>
> Hello Stanislav,
> Stored fields have nothing which findability, I believe. Usually debugQuery
> and explainOther is a right way to get what's going on there. What is $q ?
> How it's supposed to work?
>
> 24 янв. 2017 г. 18:29 пользователь "Stanislav Sandalnikov" <
> s.sandalnikov@gmail.com> написал:
>
>> Hi everyone,
>>
>> I’m facing strange Solr behavior, which could be better described in
>> examples:
>>
>>
>> With indexed but not stored IndexDate field:
>>
>> 1) With this query everything works fine, I’m getting the results back:
>> /select?fl=taskid,docid,score&q=*:*&fq=((((category:"
>> Security")))+AND+(datasource:(sites)))&fq={!frange+l%3D0}
>> query($q)&sort=IndexDate+desc&rows=100&start=0
>>
>> 2) With this query I get nothing:
>> /select?fl=taskid,docid,score&q=*:*&fq=((((category:"
>> Security")))+AND+(datasource:(sites))+AND+(IndexDate:[NOW/
>> DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&fq={!frange+l%3D0}query($q)&
>> sort=IndexDate+desc&rows=100&start=0
>>
>> Document’s IndexDate fit specified timeframe for sure.
>>
>> 3) With this query there is no category filter, but there is timeframe
>> filter and everything works fine:
>> /select?fl=taskid,docid,score&q=*:*&fq=((datasource:(sites))
>> +AND+(IndexDate:[NOW/DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&
>> fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
>>
>> Then I decided that it might be related to IndexDate as it is not stored.
>> I reindexed data with stored IndexDate field and now query number 2 works
>> just fine.
>>
>> However, I don’t get the logic here, why it doesn’t work in some
>> particular case? Can someone explain?
>>
>> Thank you in advance
>>
>> P.S. category field is indexed but not stored in all cases.
>>
>> Regards
>> Stanislav
>>
>>
>>
>>
>>
Re: Problems with stored/not-stored field filter queries
Posted by Mikhail Khludnev <gg...@gmail.com>.
Hello Stanislav,
Stored fields have nothing which findability, I believe. Usually debugQuery
and explainOther is a right way to get what's going on there. What is $q ?
How it's supposed to work?
24 янв. 2017 г. 18:29 пользователь "Stanislav Sandalnikov" <
s.sandalnikov@gmail.com> написал:
> Hi everyone,
>
> I’m facing strange Solr behavior, which could be better described in
> examples:
>
>
> With indexed but not stored IndexDate field:
>
> 1) With this query everything works fine, I’m getting the results back:
> /select?fl=taskid,docid,score&q=*:*&fq=((((category:"
> Security")))+AND+(datasource:(sites)))&fq={!frange+l%3D0}
> query($q)&sort=IndexDate+desc&rows=100&start=0
>
> 2) With this query I get nothing:
> /select?fl=taskid,docid,score&q=*:*&fq=((((category:"
> Security")))+AND+(datasource:(sites))+AND+(IndexDate:[NOW/
> DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&fq={!frange+l%3D0}query($q)&
> sort=IndexDate+desc&rows=100&start=0
>
> Document’s IndexDate fit specified timeframe for sure.
>
> 3) With this query there is no category filter, but there is timeframe
> filter and everything works fine:
> /select?fl=taskid,docid,score&q=*:*&fq=((datasource:(sites))
> +AND+(IndexDate:[NOW/DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&
> fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
>
> Then I decided that it might be related to IndexDate as it is not stored.
> I reindexed data with stored IndexDate field and now query number 2 works
> just fine.
>
> However, I don’t get the logic here, why it doesn’t work in some
> particular case? Can someone explain?
>
> Thank you in advance
>
> P.S. category field is indexed but not stored in all cases.
>
> Regards
> Stanislav
>
>
>
>
>
Re: Problems with stored/not-stored field filter queries
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/24/2017 6:15 PM, Stanislav Sandalnikov wrote:
> Thanks a lot for your valuable input. Of course you were right, the
> data was changed after reindex step, I completely forgot that
> categories are done by separate application and this application was
> pushing empty IndexDate field after update, because it couldn\u2019t
> extract a value from it since the field was not stored. By the way is
> there any way to see if there is a index for some particular field of
> a document?
As long as the field doesn't have docValues enabled, the facet feature
can be used to drill down into what terms are in the index for that
field in a document. If docValues are enabled, then you would only see
the originally indexed input, not the analyzed terms. If your field
meets this requirement, do a facet on the field, and use the q/fq
parameters to search for a single document, probably by ID.
There is also a separate program called Luke that can inspect Lucene
indexes down to the individual document level.
Thanks,
Shawn
Re: Problems with stored/not-stored field filter queries
Posted by Stanislav Sandalnikov <s....@gmail.com>.
Hi Shawn,
Thanks a lot for your valuable input. Of course you were right, the data was changed after reindex step, I completely forgot that categories are done by separate application and this application was pushing empty IndexDate field after update, because it couldn’t extract a value from it since the field was not stored. By the way is there any way to see if there is a index for some particular field of a document?
Stanislav
> 24 янв. 2017 г., в 23:43, Shawn Heisey <ap...@elyograg.org> написал(а):
>
> On 1/24/2017 8:29 AM, Stanislav Sandalnikov wrote:
>> With indexed but not stored IndexDate field:
>>
>> 1) With this query everything works fine, I’m getting the results back:
>> /select?fl=taskid,docid,score&q=*:*&fq=((((category:"Security")))+AND+(datasource:(sites)))&fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
>>
>> 2) With this query I get nothing:
>> /select?fl=taskid,docid,score&q=*:*&fq=((((category:"Security")))+AND+(datasource:(sites))+AND+(IndexDate:[NOW/DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
>>
>> Document’s IndexDate fit specified timeframe for sure.
>>
>> 3) With this query there is no category filter, but there is timeframe filter and everything works fine:
>> /select?fl=taskid,docid,score&q=*:*&fq=((datasource:(sites))+AND+(IndexDate:[NOW/DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
>
> It sounds like there are no documents that fit all three conditions --
> the correct datasource, the correct category, AND that date range.
>
>> Then I decided that it might be related to IndexDate as it is not stored. I reindexed data with stored IndexDate field and now query number 2 works just fine.
>
> Whether or not a field is stored does not affect searches, only which
> fields are returned in the results. My guess is that when you do this
> reindex, there's some difference, so different information is being sent
> to Solr and incorporated into the index. There is also the possibility
> that when you change the schema, that the changes are not limited to the
> "stored" parameter on one field. If you are changing the "type"
> parameter for the field, the effect may be larger than you realize.
>
> When you find that the search doesn't work, check the schema browser for
> allthe fields used in your query, and load the top N terms on each one.
> You may find that there aren't any terms to load for some reason, or
> that the terms that get loaded are not compatible with the part of the
> query which is being done for that field. The term info that the schema
> browser shows is not be affected by whether or not the field is stored.
> Also in the schema browser, check all the settings for those fields when
> it's working compared to when it's not working. There may be a
> difference other than the "stored" checkbox.
>
> Here's a schema browser screenshot with terms loaded from a field that
> would be compatible with your date range query:
>
> https://www.dropbox.com/s/t5b4b6hrk0wz6oz/trie-date-schema-browser.png?dl=0
>
> This screenshot comes from version 6.3.0 running in standalone mode.
>
> If everything appears to be correct on your system when it's not working
> compared to when it is working, then a bug is always a possibility, but
> a bug like that would affect a LOT of people. This list would have
> heard from those people on the problem. Such a bug should also get
> caught by the numerous tests that are part of the Lucene/Solr source
> code, tests that are frequently run by automated systems and Solr
> developers working on the code.
>
> Although your question did include more information than many do, it was
> missing a number of important details. Some of the more important
> pieces are the version of Solr and concrete information from your
> config/schema.
>
> Thanks,
> Shawn
>
Re: Problems with stored/not-stored field filter queries
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/24/2017 8:29 AM, Stanislav Sandalnikov wrote:
> With indexed but not stored IndexDate field:
>
> 1) With this query everything works fine, I\u2019m getting the results back:
> /select?fl=taskid,docid,score&q=*:*&fq=((((category:"Security")))+AND+(datasource:(sites)))&fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
>
> 2) With this query I get nothing:
> /select?fl=taskid,docid,score&q=*:*&fq=((((category:"Security")))+AND+(datasource:(sites))+AND+(IndexDate:[NOW/DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
>
> Document\u2019s IndexDate fit specified timeframe for sure.
>
> 3) With this query there is no category filter, but there is timeframe filter and everything works fine:
> /select?fl=taskid,docid,score&q=*:*&fq=((datasource:(sites))+AND+(IndexDate:[NOW/DAY-27DAYS+TO+NOW/DAY%2B1DAY]))&fq={!frange+l%3D0}query($q)&sort=IndexDate+desc&rows=100&start=0
It sounds like there are no documents that fit all three conditions --
the correct datasource, the correct category, AND that date range.
> Then I decided that it might be related to IndexDate as it is not stored. I reindexed data with stored IndexDate field and now query number 2 works just fine.
Whether or not a field is stored does not affect searches, only which
fields are returned in the results. My guess is that when you do this
reindex, there's some difference, so different information is being sent
to Solr and incorporated into the index. There is also the possibility
that when you change the schema, that the changes are not limited to the
"stored" parameter on one field. If you are changing the "type"
parameter for the field, the effect may be larger than you realize.
When you find that the search doesn't work, check the schema browser for
allthe fields used in your query, and load the top N terms on each one.
You may find that there aren't any terms to load for some reason, or
that the terms that get loaded are not compatible with the part of the
query which is being done for that field. The term info that the schema
browser shows is not be affected by whether or not the field is stored.
Also in the schema browser, check all the settings for those fields when
it's working compared to when it's not working. There may be a
difference other than the "stored" checkbox.
Here's a schema browser screenshot with terms loaded from a field that
would be compatible with your date range query:
https://www.dropbox.com/s/t5b4b6hrk0wz6oz/trie-date-schema-browser.png?dl=0
This screenshot comes from version 6.3.0 running in standalone mode.
If everything appears to be correct on your system when it's not working
compared to when it is working, then a bug is always a possibility, but
a bug like that would affect a LOT of people. This list would have
heard from those people on the problem. Such a bug should also get
caught by the numerous tests that are part of the Lucene/Solr source
code, tests that are frequently run by automated systems and Solr
developers working on the code.
Although your question did include more information than many do, it was
missing a number of important details. Some of the more important
pieces are the version of Solr and concrete information from your
config/schema.
Thanks,
Shawn