Re: Searching for an efficient and scalable way to filter query results using non-indexed and dynamic range values

[Alessandro Benedetti <a....@sease.io>]

Reading from the wiki [1]: 

" An atomic update operation is performed using this approach only when the
fields to be updated meet these three conditions:

are non-indexed (indexed="false"), non-stored (stored="false"), single
valued (multiValued="false") numeric docValues (docValues="true") fields;

the _version_ field is also a non-indexed, non-stored single valued
docValues field; and,

copy targets of updated fields, if any, are also non-indexed, non-stored
single valued numeric docValues fields. "

Now, following Diego's idea, if you model your data as a single valued
incremental numeric field, you should not index it.
This can be a blocker, but I remember that even if you don't index a field,
it is possible to search on it using the docValues data structure.
But I have no idea if the performance are going to be acceptable in your
case.
IntPoint seems compatible with docValues [2]:

"Int|Long|Float|Double|Date PointField

If the field is single-valued (i.e., multi-valued is false), Lucene will use
the NUMERIC type."

I would probably give it a shot.
But is it compatible with what you need ? Is the incremental crawling cycle
good enough ? don't you need to associate the Crawling cycle to the
Experiment Id ?

Cheers



[1]
https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#UpdatingPartsofDocuments-In-PlaceUpdates
[2] https://lucene.apache.org/solr/guide/6_6/docvalues.html



-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Searching for an efficient and scalable way to filter query results using non-indexed and dynamic range values

[Emir Arnautović <em...@sematext.com>]

Hi,
I did not check it in code, but based on earlier comments on ML, it seems that in place updates are not as it sounds - it will rewrite doc values for the segment that is updated. If you really want to avoid index changes, you can maybe use external field: https://lucene.apache.org/solr/guide/6_6/working-with-external-files-and-processes.html <https://lucene.apache.org/solr/guide/6_6/working-with-external-files-and-processes.html> and function queries when evaluating.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 1 Feb 2018, at 11:10, Alessandro Benedetti <a....@sease.io> wrote:
> 
> Reading from the wiki [1]: 
> 
> " An atomic update operation is performed using this approach only when the
> fields to be updated meet these three conditions:
> 
> are non-indexed (indexed="false"), non-stored (stored="false"), single
> valued (multiValued="false") numeric docValues (docValues="true") fields;
> 
> the _version_ field is also a non-indexed, non-stored single valued
> docValues field; and,
> 
> copy targets of updated fields, if any, are also non-indexed, non-stored
> single valued numeric docValues fields. "
> 
> Now, following Diego's idea, if you model your data as a single valued
> incremental numeric field, you should not index it.
> This can be a blocker, but I remember that even if you don't index a field,
> it is possible to search on it using the docValues data structure.
> But I have no idea if the performance are going to be acceptable in your
> case.
> IntPoint seems compatible with docValues [2]:
> 
> "Int|Long|Float|Double|Date PointField
> 
> If the field is single-valued (i.e., multi-valued is false), Lucene will use
> the NUMERIC type."
> 
> I would probably give it a shot.
> But is it compatible with what you need ? Is the incremental crawling cycle
> good enough ? don't you need to associate the Crawling cycle to the
> Experiment Id ?
> 
> Cheers
> 
> 
> 
> [1]
> https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#UpdatingPartsofDocuments-In-PlaceUpdates
> [2] https://lucene.apache.org/solr/guide/6_6/docvalues.html
> 
> 
> 
> -----
> ---------------
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html