You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by gnandre <ar...@gmail.com> on 2022/03/29 02:25:51 UTC
How to sort Solr search results by hour/day/week/month/year views
I have a search interface that searches for videos. There is a filter there
that can sort the results based on views for last hour, last day,
last week, last month and last year. Is there a way to achieve this sorting
without always needing to update the index? E.g. After each specific
duration expires, all documents need to be indexed again for newly
calculated values.This is too much indexing overload. (p.s I am doing
atomic indexing for these fields already)
Re: How to sort Solr search results by hour/day/week/month/year views
Posted by gnandre <ar...@gmail.com>.
Thanks all for the helpful information!
On Tue, Mar 29, 2022 at 11:32 AM Geren White <ge...@1stdibs.com> wrote:
> Make sure to set up the ExternalFileFieldReloader described here:
>
> https://solr.apache.org/guide/6_6/working-with-external-files-and-processes.html#WorkingwithExternalFilesandProcesses-ReloadinganExternalFile
> .
> This caches the file values in solr so that you don't do a read everytime.
> Also make sure that the file contents are sorted by id otherwise the lookup
> can be very slow.
>
> On Tue, Mar 29, 2022 at 4:33 AM Sidharth Negi <si...@gmail.com>
> wrote:
>
> > Are you aware about how this external file is integrated with the index?
> >
> > My index consists of ~25 million docs and I'm asking this because I'm
> > concerned about whether or not a disk lookup from a separate file will
> > cause increased latency in my Solr query (which uses a few of such daily
> > updating fields within boost function query).
> >
> > On Tue, Mar 29, 2022 at 9:37 AM Walter Underwood <wu...@wunderwood.org>
> > wrote:
> >
> > > You could put the view counts in an external file field. That is what
> > that
> > > was designed for.
> > >
> > > Making it work in a Solr Cloud cluster isn’t very clean. You’ll need to
> > > update the file on each node. But it should work. It has been a while
> > since
> > > I’ve used it.
> > >
> > >
> > >
> >
> https://solr.apache.org/guide/8_11/working-with-external-files-and-processes.html
> > >
> > > wunder
> > > Walter Underwood
> > > wunder@wunderwood.org
> > > http://observer.wunderwood.org/ (my blog)
> > >
> > > > On Mar 28, 2022, at 7:25 PM, gnandre <ar...@gmail.com>
> wrote:
> > > >
> > > > I have a search interface that searches for videos. There is a filter
> > > there
> > > > that can sort the results based on views for last hour, last day,
> > > > last week, last month and last year. Is there a way to achieve this
> > > sorting
> > > > without always needing to update the index? E.g. After each specific
> > > > duration expires, all documents need to be indexed again for newly
> > > > calculated values.This is too much indexing overload. (p.s I am doing
> > > > atomic indexing for these fields already)
> > >
> > >
> >
>
>
> --
> *Geren White | Senior Director, Engineering*
> *(e)* geren@1stdibs.com
>
Re: How to sort Solr search results by hour/day/week/month/year views
Posted by Geren White <ge...@1stdibs.com>.
Make sure to set up the ExternalFileFieldReloader described here:
https://solr.apache.org/guide/6_6/working-with-external-files-and-processes.html#WorkingwithExternalFilesandProcesses-ReloadinganExternalFile.
This caches the file values in solr so that you don't do a read everytime.
Also make sure that the file contents are sorted by id otherwise the lookup
can be very slow.
On Tue, Mar 29, 2022 at 4:33 AM Sidharth Negi <si...@gmail.com>
wrote:
> Are you aware about how this external file is integrated with the index?
>
> My index consists of ~25 million docs and I'm asking this because I'm
> concerned about whether or not a disk lookup from a separate file will
> cause increased latency in my Solr query (which uses a few of such daily
> updating fields within boost function query).
>
> On Tue, Mar 29, 2022 at 9:37 AM Walter Underwood <wu...@wunderwood.org>
> wrote:
>
> > You could put the view counts in an external file field. That is what
> that
> > was designed for.
> >
> > Making it work in a Solr Cloud cluster isn’t very clean. You’ll need to
> > update the file on each node. But it should work. It has been a while
> since
> > I’ve used it.
> >
> >
> >
> https://solr.apache.org/guide/8_11/working-with-external-files-and-processes.html
> >
> > wunder
> > Walter Underwood
> > wunder@wunderwood.org
> > http://observer.wunderwood.org/ (my blog)
> >
> > > On Mar 28, 2022, at 7:25 PM, gnandre <ar...@gmail.com> wrote:
> > >
> > > I have a search interface that searches for videos. There is a filter
> > there
> > > that can sort the results based on views for last hour, last day,
> > > last week, last month and last year. Is there a way to achieve this
> > sorting
> > > without always needing to update the index? E.g. After each specific
> > > duration expires, all documents need to be indexed again for newly
> > > calculated values.This is too much indexing overload. (p.s I am doing
> > > atomic indexing for these fields already)
> >
> >
>
--
*Geren White | Senior Director, Engineering*
*(e)* geren@1stdibs.com
Re: How to sort Solr search results by hour/day/week/month/year views
Posted by Sidharth Negi <si...@gmail.com>.
Are you aware about how this external file is integrated with the index?
My index consists of ~25 million docs and I'm asking this because I'm
concerned about whether or not a disk lookup from a separate file will
cause increased latency in my Solr query (which uses a few of such daily
updating fields within boost function query).
On Tue, Mar 29, 2022 at 9:37 AM Walter Underwood <wu...@wunderwood.org>
wrote:
> You could put the view counts in an external file field. That is what that
> was designed for.
>
> Making it work in a Solr Cloud cluster isn’t very clean. You’ll need to
> update the file on each node. But it should work. It has been a while since
> I’ve used it.
>
>
> https://solr.apache.org/guide/8_11/working-with-external-files-and-processes.html
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
> > On Mar 28, 2022, at 7:25 PM, gnandre <ar...@gmail.com> wrote:
> >
> > I have a search interface that searches for videos. There is a filter
> there
> > that can sort the results based on views for last hour, last day,
> > last week, last month and last year. Is there a way to achieve this
> sorting
> > without always needing to update the index? E.g. After each specific
> > duration expires, all documents need to be indexed again for newly
> > calculated values.This is too much indexing overload. (p.s I am doing
> > atomic indexing for these fields already)
>
>
Re: How to sort Solr search results by hour/day/week/month/year views
Posted by Walter Underwood <wu...@wunderwood.org>.
You could put the view counts in an external file field. That is what that was designed for.
Making it work in a Solr Cloud cluster isn’t very clean. You’ll need to update the file on each node. But it should work. It has been a while since I’ve used it.
https://solr.apache.org/guide/8_11/working-with-external-files-and-processes.html
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Mar 28, 2022, at 7:25 PM, gnandre <ar...@gmail.com> wrote:
>
> I have a search interface that searches for videos. There is a filter there
> that can sort the results based on views for last hour, last day,
> last week, last month and last year. Is there a way to achieve this sorting
> without always needing to update the index? E.g. After each specific
> duration expires, all documents need to be indexed again for newly
> calculated values.This is too much indexing overload. (p.s I am doing
> atomic indexing for these fields already)